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OFFICE  OF  THE  SECRETARY  OF  DEFENSE 

3140  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-3140 


DEFENSE  SCIENCE 
BOARD 


MEMORANDUM  FOR  UNDER  SECRETARY  OF  DEFENSE  (ACQUISITION, 

TECHNOLOGY  &  LOGISTICS) 

SUBJECT:  Final  Report  of  the  Defense  Science  Board  (DSB)  Task  Force  on  Missile 
Defense  Phase  III  -  Modeling  and  Simulation 

I  am  pleased  to  forward  the  final  report  of  the  DSB  Task  Force  on  Missile 
Defense  Phase  III — Modeling  and  Simulation.  This  report  evaluates  the  state  of 
modeling  and  simulation  (M&S)  efforts  supporting  the  Ballistic  Missile  Defense  System 
and  makes  recommendations  for  improving  future  M&S  development. 

Modeling  and  simulation  plays  a  unique  and  central  role  in  the  development  of 
the  missile  defense  program.  Effective  modeling  and  simulation  is  the  sole  means  to 
assess  overall  system  performance  and  quality,  train  and  operate  at  the  system  level,  and 
look  beyond  current  system  plans  and  programs.  The  task  force  addresses  these  critical 
features  and  also  recommends  methods  to  improve  confidence  in  the  validation  and 
accreditation  of  the  various  levels  of  models  and  simulations. 

I  endorse  all  of  the  Task  Force’s  recommendations  and  propose  you  review  their 

report. 


William  Schneider,  Jr. 
DSB  Chairman 
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MEMORANDUM  FOR  THE  CHARIMAN,  DEFENSE  SCIENCE  BOARD 

SUBJECT:  Report  of  the  Defense  Science  Board  Task  Force  on  Ballistic  Missile 
Defense  Phase  III:  Modeling  and  Simulation 


Modeling  and  simulation  (M&S)  plays  a  key  role  in  both  developing  the 
Ballistic  Missile  Defense  System  and  assessing  its  likely  effectiveness.  M&S  for 
missile  defense  is  an  expansive  undertaking,  ranging  from  detailed  engineering 
simulations  of  individual  components  to  system-of-system  level  simulations. 
The  Defense  Science  Board  Task  Force  was  asked  to  assess  five  specific  areas 
involving  modeling  and  simulation  for  the  missile  defense  program: 

•  The  scope  of  the  modeling  and  simulation  effort 

•  The  appropriateness  of  the  level  of  fidelity 

•  The  impact  of  communication  on  the  end-to-end  models 

•  The  approaches  to  the  validity  of  simulations 

•  Additional  opportunities  for  M&S  contributions 

Flight  tests  for  missile  defense  systems  in  today's  environment  require 
substantial  time,  resources,  and  expertise.  Much  more  than  in  earlier  missile 
defense  programs,  today  modeling  and  simulation  must  be  relied  upon  to 
account  for  all  pieces  of  the  program,  the  resources  that  go  into  all  those  pieces, 
and  the  integration  of  the  program  elements  into  a  single  system-of-systems. 
Thus,  modeling  and  simulation  is  a  critical  part  of  the  overall  missile  defense 
program. 

To  assure  that  modeling  and  simulation  fulfills  its  role,  the  Task  Force  makes 
the  following  bottom  line  findings  and  recommendations,  which  are  addressed 
in  detail  in  the  body  of  the  report. 

•  Missile  defense  models  and  simulations  are  needed  to  provide 
analytical  underpinnings  for  system  design,  development,  and 
operations.  To  perform  this  role,  the  many  individual  models  and 


simulations  need  to  be  more  adequately  linked  and  integrated,  at  the 
system  level,  with  appropriate  levels  of  fidelity. 

•  Missile  defense  development,  deployment,  and  testing  must  be 
supported  by  adequate  system  level  models.  The  Missile  Defense 
Agency  (MDA)  should  specify  the  system-level  model  that  will  be 
used  for  system  effectiveness  assessment  and  require  that  all  elements 
in  the  program  provide  appropriate  models  to  interface  with  the 
system-level  model.  The  linkages  that  currently  exist  between  models 
do  not  provide  the  needed  insight  to  support  system-level 
development,  deployment,  and  testing. 

•  A  significant  modeling  and  simulation  program  needs  to  be  oriented 
to  support  future  development  of  the  Ballistic  Missile  Defense  System 
at  the  block  level.  M&S  representations  should  be  developed  and 
delivered  for  each  block  upgrade. 

•  Improvements  are  needed  in  the  verification,  validation  and 
accreditation  processes  for  missile  defense  models  and  simulations. 
Ownership  of  the  verification  and  validation  and,  separately,  of  the 
accreditation  processes  at  a  senior  level  should  be  established.  A 
primary  responsibility  for  this  individual  is  to  obtain  outside  expertise 
to  provide  MDA  with  increased  confidence  in  block  model 
representations,  given  the  complexity  of  the  missile  defense  system. 

•  To  provide  adequate  management  attention  to  the  critical  role  of 
modeling  and  simulation,  MDA  should  establish  a  Modeling  and 
Simulation  Executive  within  its  organization,  reporting  to  the  Director. 
This  individual  should  be  responsible  for  funding,  acceptance, 
delivery,  and  maintenance  of  the  M&S  software  for  each  component, 
element,  and  block. 

By  implementing  the  recommendations  contained  in  this  report,  and 
summarized  here,  the  MDA  can  strengthen  its  current  modeling  and  simulation 
effort  to  meet  the  needs  of  such  a  complex  and  important  program. 


General  I/arry  Welch,  USAF  (ret.)  Dr.  William  Graham 
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Executive  Summary 


Executive  Summary 


Terms  of  Reference 


■  Phase  I  -  Reported  in  May  2002  -  Assess  the  program  for 
mid-course  discrimination 

■  Phase  II  -  Reported  in  October  2002  -  examine: 

-  Capability  need  (evolution  of  ballistic  missile  threats) 

-  International  cooperation 
“  Boost  phase  technology 

-  Battle  management,  command  and  control,  and  communications 

■  Phase  III  -  Modeling  and  Simulation  -  assess: 

-  The  scope  of  the  modeling  and  simulation  effort 

-  The  appropriateness  of  the  level  of  fidelity 

--  The  Impact  of  communication  on  the  end-to-end  models 
“  The  approaches  to  the  validity  of  simulations 
“  Additional  opportunities  for  M&S  contribution 


This  report  covers  Phase  III  of  the  work  of  the  Defense  Science 
Board  (DSB)  Task  Force  on  Missile  Defense.  Phase  I  examined  mid¬ 
course  discrimination.  Phase  II,  a  2002  DSB  Summer  Study,  examined 
the  four  issues  shown  in  the  chart  above.  Following  completion  of  the 
Phase  II  report,  the  Director,  Missile  Defense  Agency  (MD A)  asked 
the  Task  Force  to  examine  modeling  and  simulation  (M&S).  The 
Under  Secretary  Defense  for  Acquisition,  Technology  and  Logistics 
(USD[AT&L])  concurred.  The  request  was  further  refined  to  the  five 
areas  shown.  After  a  brief  introduction,  this  report  will  cover  each  of 
the  five  issues  in  turn. 
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Missile  Defense  Phase  III 


Executive  Summary, 


This  chart  is  a  reminder  of  the  scope  of  the  effort  that  modeling 
and  simulation  must  serve.  The  end  capability  required  is  a  capability 
to  defeat  missiles  of  all  ranges  in  all  phases  of  flight.  This  capability  is 
to  be  achieved  with  spiral  development— fielding  increments  of 
capability  that  build  toward  the  end  objective. 
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Executive  Summary 


Bottom  Lines 


■  Credible  modeling  and  simulation  (M&S)  is  essential  to 
development,  testing  and  deployment.  It  provides  the  only 
way/place  that  can: 

-  Account  for  the  how  &  why  of  full  program  spending 
•  -  Integrate  all  the  parts 

-  Solve  element  problems  at  the  architecture  level 

-  Test  at  the  architecture/integrated  system  level 

-  Assess  Ballistic  Missile  Defense  (BMD)  integrated  system  quality 
and  reliability 

-  Explore  the  full  operating  envelope 

-  Look  beyond  current  plans  and  programs 

-  Demonstrate  to  the  user/operator  how  the  system  is  expected  to 
perform  in  potential  scenarios 

-  Train  and  operate  at  the  system  level 


This  chart  and  the  five  that  follow  offer  the  bottom  line  findings 
and  recommendations  of  the  Task  Force.  Subsequent  sections  will 
provide  the  detail  underwriting  these  bottom  lines.  This  first  chart 
provides  a  broad  picture  of  the  demands  on  modeling  and  simulation 
to  support  MDA  development,  acquisition,  and  operation.  Given  the 
nature  of  the  missile  defense  program,  M&S  plays  an  unusually 
central  role.  As  indicated,  it  is  the  only  way  to  account  for  all  the 
pieces  of  the  program,  the  resources  that  go  into  all  those  pieces,  and 
their  integration  into  elements  (e.g.,  ground-based  missile  defense, 
kinetic  boost  phase.  Aegis,  and  PAC-3)  and  into  the  system  of 
systems.  It  is  the  only  way  to  assess  overall  performance  and  quality. 
M&S  is  also  the  only  way  to  train  and  operate  at  the  system  level.  The 
remaining  bottom  lines  charts  are  self-explanatory. 
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Executive  Summary  _ 


Bottom  Lines  (cont.) 

■  To  provide  needed  analytical  underpinnings  for  system  design, 
development,  and  operations,  MDA  models  and  simulations  need  to 
be  more  adequately  linked  and  integrated  with  appropriate  fidelity 

-  The  specific  purposes  for  the  various  levels  of  modeling  and  simulation 
(from  component  to  campaign)  need  to  be  explicitly  and  authoritatively 
defined 

The  key  MDA  models  and  simulations  are  legacy  models,  developed 
largely  as  stand-alone  models  and  hence  are  not  well  designed  to  fit 
together  into  the  needed  modeling  and  simulation  system  architectures 

-  Key  models  in  the  family  of  MDA  models  and  simulations  need  to  be 
vertically  integrated  -  that  is  validated,  accredited  component  and 
element  models  need  to  logically  feed  system  level  models  that,  in  turn, 
feed  campaign  and  theater  models. 

-  Element  models  need  to  be  horizontally  and  seamlessly  linked  and 
validated  for  their  intended  purpose  to  provide  usable  system  level 
models 

-  The  fidelity  and  credibility  of  the  element  and  component  representations 
in  the  system  level  models  are  not  adequate  to  provide  confidence  in 
model  outputs 


Bottom  Lines  (cont.) 


■  MDA  development,  deployment  and  testing  must  be  supported  by 
adequate  system  level  models 

-  The  national  teams  (Battle  Management  Command,  Control  and 
Communications  [NTB]  and  Systems  Engineering  &  Integration  [NTS]) 
are  developing  separate  system  level  models  with  little  communication  to 
determine  whether  a  common  mode!  would  be  a  more  effective  solution 

-  “Systems”  and  command  and  control,  battle  management,  and 
communications  {C2/BM/C)  models  need  to  be  developed  in  close 
collaboration  to  exploit  multi-sensor  data 

-  Integrated  discrimination,  designation  emphasis  is  needed  to  ensure  that 
the  timelines,  latency,  and  message  content  provide  for  needed  data 
fusion  in  the  system  design  and  system  simulations 

-  The  MDA  should  specify  the  system  level  model  to  be  used  for  system 
effectiveness  assessment  and  require,  contractually,  that  all  elements 
provide  interfacing  representations.  The  Task  Force  found  MDWAR 
promising  though  not  yet  adequate  to  the  task. 

-  The  M&S  focus  needs  to  shift  from  continuing  to  expand  currently  largely 
adequate  element  models  to  filling  some  glaring  gaps  (e.g.,  the 
discrimination  function)  and  on  integrated  system  level  M&S 

-  The  simulation  architectures  do  not  allow  the  models  and  simulations  to 
interact  as  needed  to  provide  the  desired  Insights 
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Executive  Summary 


Bottom  Lines  (cont.) 


■  Support  of  future  development 

-  There  needs  to  be  a  significant  M&S  program  specifically  oriented 
to  defined  and  future  conceptual  development  blocks 

-  A  system  level  (Missile  Defense  Warfare  Assessment  and 
Research  Simulation  [MDWAR]?)  M&S  deliverable  should  be 
developed  for  each  block  with  appropriate  configuration  control  to 
assure  the  user/operator  that  the  representation  is  valid  and 
credible  for  envisioned  scenarios 

-  The  system  level  modeling  requirements  need  to  also  serve  the 
needs  of  United  States  Strategic  Command  (USSTRATCOM)  in 
filling  their  global  Integrated  missile  defense  responsibilities 


Bottom  Lines  (cont.) 


■  Validation  and  accreditation 

“  Given  the  complexity  of  the  integrated  missile  defense  task  and 
the  limitations  on  validation  through  flight  testing,  hardware-ln-the- 
loop  (HWIL)  simulations  become  key  to  confidence  in  M&S  validity 

-  A  master  "Accreditation  Tree”  is  needed  with  clearly  delineated 
buyoff  dates  by  M&S  users,  fully  linked  to  the  system  development 
and  deployment  schedule 

-  There  is  a  need  for  an  authoritative  library  of  accredited  models  at 
levels  from  phenomenology  and  physics  to  integrated  system  level 
models.  Accreditation  documentation  should  be  specific  as  to  the 
purposes  for  which  accredited 
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Executive  Summary  , 


Bottom  Lines  (cont.) 

■  A  Modeling  and  Simulation  Executive  in  MDA,  reporting  to  the 
Director,  Is  needed  to  provide  adequate  management  attention  to  the 
critical  role  of  Modeling  and  Simulation 

-  Responsible  for  the  funding,  acceptance,  delivery,  and  maintenance  of 
the  M&S  software  for  each  component,  element,  and  block 

-  Establish  MDA  ownership  of  Systems  and  C2/BM/C  M&S  development  at, 
senior  level  with  responsibility,  authority,  and  budget  to  provide  coherent 
product  for  the  user/operator 

■  Establish  (separate)  MDA  ownership  of  the  independent  verification, 
validation,  and  accreditation  (W&A)  process  at  a  senior  level  with 
sufficient  resources  to  engage  external,  rather  than  just  internal, 
expertise  to  provide  MDA  and  the  user/operator  a  confident 
representation  of  Block  system  performance  (through  assuring  that 
each  element  of  the  M&S  representation  is  assessed). 

-  Establish  a  W&A  Quality  Assessment/Audit  Team  with  a  balance  of 
experienced  missile  defense  element  algorithm,  C2/BM/C,  software 
experts,  and  operators  that  reports  to  the  M&S  Executive 
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Chapter  1 


Chapter  1.  Scope  of  The  Modeling  and  Simulation 
Effort 

This  chapter  discusses  the  scope  of  the  DSB  study.  The  Task  Force 
considered  overall  adequacy  to  be  a  part  of  scope  and  thus  includes 
findings  and  recommendations  on  the  overall  adequacy  and 
suitability  of  current  M&S  and  development  plans. 


Scope  of  Simulation  Needs 

■  Support  program  decision  points,  design  trades 

■  Predict  &  assesses  system  performance 

■  Help  develop  C2/BM/C  capabilities  specification 

■  Help  define  BMDS  spiral  and  technology  development  needs 

■  Support  detailed  engineering  development 

■  Explore  concepts  and  system  trade-offs 

■  Guide  and  augment  flight  testing 

•  Support  exercises,  wargames,  training 

■  Identify  block  system  capabilities  against  ballistic  missiles  of  all 
ranges  in  all  phases  of  flight  for  the  user/operator 

■  Support  operational  decisions  In  execution 

It  is  critically  important  to  define  the  purpose  of 
each  level  of  simulation  to  guide  models  and 
simulation  development 


The  scope  required  for  the  Ballistic  Missile  Defense  System 
(BMDS)  development  blocks  is  determined  by  the  backgrounds, 
targets,  elements,  activities,  and  systems  that  M&S  must  describe 
adequately  to  assess  overall  performance  and  contributions  of  the 
major  elements. 

Overall  issues  include: 

■  The  range  of  threats  and  defensive  systems  that  can 
be  analyzed 
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Scope  of  the  Modeling 
AND  Simula  tion  Effor  t 


■  Ability  to  model  relevant  threats  and  elements  for 
initial  defensive  operations  (IDO)  and  subsequent 
blocks 

■  Adequacy  of  the  scope  to  treat  IDO  and  subsequent 
blocfe;  complicated  in  that  the  systems  are  still  to  be 
defined 

■  The  scope  of  M&S  needed  for  adequate  analyses 

■  The  more  specific  issues  listed  in  the  chart  above  are 
taken  from  the  summary  MDA  briefing  at  the  Joint 
National  Integration  Center  (JNIC),  which  defined 
the  specific  elements  that  implicitly  define  a 
program  of  adequate  scope 
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Chapter  1 


Pyramid  Approach  to  the  Scope  of  BMDS 
Models  and  Simulations  (M&S) 


Validity  Requires 
Vertical  integration 


/  CampaignX 

uires  /  &  Theater  \ 

/  Models  \ 

/  System  Level  \ 

/  Models  (End-to-End)  \ 

/  BMDS  Element  M&S  \ 

/  Component  Models  ' 

Core  Phenomenology  Models 
Verification  -  Validation  -  Accreditatidh 


Decreasing  Fidelity 
Acceptable 


The  MDA  M&:S  community  makes  frequent  reference  to  a 
pyramid  of  models  of  increasing  scope  and  reducing  fidelity.  In 
actual  practice,  the  models  are  not  integrated  in  horizontal  layers  and 
there  is  little  linkage  to  models  at  higher  level  of  aggregation  from 
those  below  in  the  pyramid.  These  shortcomings  and  their 
consequences  are  discussed  furtlier  in  following  charts  and  sections. 
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Scope  of  the  Modeling 
AND  Simula  tion  Effort 


This  chart  depicts  the  MD  A  notional  relationship  between  the 
system  and  element  models  and  between  models  within  the  vertical 
layers.  However,  as  noted  earlier,  the  current  core  models  have  little 
or  no  horizontal  integration.  Thus,  the  evaluation  of  the  elements' 
separate  and  integrated  benefits  is  uncertain.  The  system  level 
models  do  not  adequately  address  the  C2/BM/ C  interface  and  data 
flow. 
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The  Heritage  and  Challenge 

■  Most  current  MDA  M&S  activities  are  legacy  items 

-  Developments,  in  some  cases,  more  than  a  decade  old 

~  Many  activities  originally  dominated  by  needs  and  Interests  of  the 
MDAPs 

-  Developed  under  different  organizations,  in  isolation,  with  different 
objectives 

■  The  MDA  objective  to  treat  M&S  as  a  coordinated  effort  is 
only  a  few  years  old 

-  Focus  is  primarily  at  the  phenomena  and  integrated  element  level 
with  other  hierarchical  levels  mostly  being  developed 
autonomously 

-  A  major  part  of  the  strategy  and  effort  and  a  major  challenge  is 
salvaging/converting/migrating/extending  legacy  items 

-  A  major  issue  is  the  appropriate  allocation  of  resources  in 
achieving  the  required  capability  across  the  M&S  classes  and 
elements 


There  has  been  a  great  deal  of  activity  in  the  Department  of 
Defense  (DoD)  attempting  to  integrate  M&S  into  programs  and 
focusing  on  the  issue  of  fidelity  for  performance  assessment, 
acquisition  support,  and  training  and  war  games.  Most  current 
MDA-specific  M&S  activities  are  legacy  items,  sometimes  a  decade  or 
more  old,  and  originally  developed  with  different  objectives  than 
those  now  required  by  MDA.  These  models  are  being  updated, 
although  plans  to  use  the  modernized  versions  are  not  clear. 
Moreover,  there  appear  to  be  many  redundant  codes  that  are 
maintained  for  a  variety  of  non-technical  reasons.  An  organized 
attempt  to  coordinate  MDA  M&S  activity  and  resources  has  only 
been  in  place  for  a  few  years  and  has  focused  primarily  on  managing 
the  phenomena-,  core-,  or  integration-class  simulations. 
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Scope  of  the  Modeling 
AND  Simula  tion  Effort 


Element,  Activity  &  Integration  Model  Shortfalls 

•  Campaign  and  theater  models  -  CAPS  &  JAWARS  models 

-  Not  integrated 

-  Not  linked  to  the  models  below  them  in  the  M&S  hierarchy 

-  Little  contribution  to  increased  scope 

•  System  level  models 

-  MDWAR  is  a  widely  used  MDA  system  model 

■  BMDS  model  focus  is  largely  on  C2/BM/C 

■  Available  element  models  appear  appropriate,  but  decoupled 

-  Models  essential  for  later  phases  not  available  (boost  phase, 
discrimination) 

■  Important  candidate  future  elements  are  not  available  (e.g.,  Space 
Tracking  and  Surveillance  System  [STSS],  space-based  interceptor, 
alternative  Exoatmospheric  Kill  Vehicle  [EKV]) 

■  Threat,  phenomenology  models  lack  essential  data  (e.g. .hypervelocity 
Impact) 

■  MDA  needs  to  ensure  that  core  phenomenology  models  are  managed 
and  maintained  to  ensure  availability  of  valid  Inputs  for  the  BMDS 
M&S  community 


As  noted  earlier,  campaign-  Commanders  Analysis  Planning  and 
Simulation  (CAPS)  and  theater-level  Joint  Theater  Warfighting 
System  (JTWAR)  models  are  currently  decoupled  from  those  below 
them  in  the  M&S  pyramid.  They  do  not  explicitly  guide  or  draw  on 
the  results  of  MDWAR  level  simulations.  They  use  independent, 
shorthand  versions  of  engagements  that  are  fast  and  convenient  to 
run,  but  that  do  not  provide  confidence  in  the  validity  of  the  insights. 

Force-on-force,  mission,  and  system  integration  level  models  are 
redundant,  do  not  treat  distinct  levels  of  aggregation,  and  are  not 
closely  coupled  to  element  level  models.  They  contain  similar  but 
independent  aggregated  models  of  elements  and  communication, 
which  have  not  been  extensively  compared. 

MDWAR  appears  capable  of  adequate  scope  and  extension  to  the 
level  of  sensors,  interceptors,  communications,  and  command  control 
(C2)  needed  for  near-term  system  level  studies.  Even  so,  it  currently 
has  significant  limitations  discussed  on  the  next  chart. 

Ballistic  missile  defense  activity-level  models  are  immature  and 
decoupled.  Only  the  battle  management  and  communications  (BMC) 
has  significant  coupling,  activity,  and  potential  impact;  but  it  is 
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incomplete.  Progress  is  hindered  by  continuing  redefinition  of  its 
scope. 

Element  level  models  seem  individually  adequate  but  are 
decoupled  from  one  another.  Important  models  needed  for  later 
phases— such  as  surface-  and  space-based  boost  phase  intercept 
(BPI)  — are  not  now  available,  altliough  some  are  being  built. 

Threat,  phenomenology,  and  lethality  models  are  flexible,  but 
they  lack  data  of  the  requisite  fidelity  in  the  regimes  of  interest  for 
IDO  and  later  phases.  Some  data  such  as  hypervelocity  impact  in 
midcourse  and  boost  cannot  be  measured  in  the  laboratory  or  field 
and  cannot  be  credibly  deduced  from  system-level  tests.  Planned 
improvements  will  make  data  more  accessible,  but  not  necessarily 
more  credible. 
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Scope  OF  the  Modeling 
AND  Simula  ijon  Effort 


MDWAR  -  A  Candidate  System  Level 
(End-to-End)  Model 

■  Modern,  object-oriented,  event  driven  model  architecture 

-  Variable  fidelity,  messages,  N-1  element  simulations 

-  Range  of  variable  fidelity  models 
“  Causal  for  correct  system  delays 

-  Insert  hardware,  models  thru  gateway — interfaced  w/  MDSE 

■  Sensor  models  tested  against  other  systems  models 

-  Not  against  detailed  element  models  with  some  exceptions 

-  Element  program  offices  reluctant  to  provide  representations  for 
MDWAR  or  to  support  comparison  of  outputs 

■  Configuration  control  via  complicated  library 

■  Need  to  examine,  in  detail,  how  and  when  to  incorporate 
needed  extensions 


MDWAR  is  a  flexible,  event-driven  model  with  modern,  object- 
oriented  programming,  in  which  events  are  modeled  in  a  causal 
manner  if  the  relevant  system  delays  are  specified  correctly  by  the 
user.  At  present,  MDWAR  has  important  strengths  and  weaknesses. 
It  has  a  range  of  models  of  sensors,  interceptors,  and  communications 
of  variable  fidelity  that  seem  adequate  to  provide  element 
simulations  of  the  components.  Its  suite  of  simplified  radar  and 
infrared  sensor  models  have  been  tested  against  those  in  Extended 
Air  Defense  Simulation  (E  ADSIM)  and  other  systems  models  of 
similar  fidelity  and  have  produced  reasonable  agreement  on 
detection,  track,  commit,  and  intercept  times.  However,  these  models 
have  not  been  tested  against  detailed  element  models. 

MDWAR  explicitly  treats  actual  tactical  and  strategic  message 
formats.  It  treats  their  transmission,  collisions,  and  delays 
stochastically.  It  allows  for  the  federation  of  other  communication 
models  and  hardware  thru  a  flexible  JRE-type  gateway  to  federate 
with  Missile  Defense  System  Exerciser  (MDSE)  to  support  HWIL 
simulations. 

MDWAR  uses  Ground-based  Midcourse  Defense  (GMD)- 
supplied,  pre-competitive  fly-out  tables  for  the  ground-based 
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intercept  (GBI)  and  exoatmospheric  kill  vehicle  (EKV)  whose 
grounding  in  and  comparison  with  data  is  unclear.  The  treatment  of 
communication,  interceptors,  and  C2  is  deficient. 

Code  versions  and  results  are  configuration  controlled  through  a 
detailed  library  of  code,  input,  and  output  conditions.  This  library 
could  be  used  to  transmit  appropriate  versions  of  MDWAR  to  users; 
but  the  current  version  is  complicated,  requiring  about  800,000 
inputs. 

MDWAR  uses  the  Jet  Propulsion  Laboratory  (JPL)-developed 
SPEEDES  operating  system  for  distributed  processing  to  achieve 
running  times  that  are  typically  several  times  faster  than  real  time  to 
support  man-in- the-loop  simulations.  SPEEDES  is  a  flexible  system 
that  is  appropriate  for  distributing  processing  over  10s  of 
intermediate-level  work  stations.  There  are  other  ways  of  distributing 
such  simulations  over  many  and  faster  CPUs,  some  of  them  are 
compatible  with  JAVA  and  dynamic  allocation.  SPEEDES  is  not. 
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Some  Needed  Contributions  to  System  Engineering 

■  Specification  compliance  issue 

-  Adequacy  of  the  baseline  architecture  to  satisfy  the  system 
capability 

■  Sensitivity  and  breakpoint  analysis 

■  Cost  effectiveness  trades 

-  Tradeoffs  between  operational  and  system  investments  to 
minimize  cost 

-  The  most  cost  effective  flowdown  and  allocation  of  specifications 

-  The  point  of  diminishing  returns  for  BMDS  investment 


MDA  has  identified  the  main  areas  where  M«&:S  needs  to 
contribute  more  to  systems  engineering.  The  systems-level  models 
now  in  development,  particularly  the  extension  of  MDWAR  to  a 
C2/BM/C  model,  could  address  some  of  these  needs,  although 
missing  models  and  lack  of  calibration  currently  undercuts  their 
potential  contribution.  The  models  are  not  sufficiently  tested  to 
support  sensitivity  or  breakpoint  analyses  or  cost-effectiveness 
trades.  The  lack  of  key  models  for  boost  phase  and  discrimination 
limits  their  ability  to  guide  investment  planning  or  acquisition. 
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Findings 

■  Campaign  &  theater  level  models  -  CAPS  and  JWARS 

-  Not  linked  to  the  models  that  need  to  provide  the  inputs 

-  Of  limited  value  for  such  purposes  as  offense/defense  assessment 
and  trade-offs 

■  There  is  a  family  of  system  level  models  capable  of 
operating  at  various  levels  of  fidelity 

-  Models  are  redundant  &  decoupled 

-  Not  linked  to  element  models 

-  Element  representations  in  these  models  are  not  provided  by  the 
element  program  offices 

-  Validity  and  accreditation  Is  less  than  robust 

-  No  evidence  that  M&S  used  in  coherent  fashion  to  support  needs 


Theater-  (JWARS)  and  campaign-level  (CAPS)  models  are  over 
simplified  and  decoupled  from  system  level  and  element  models. 
CAPS  and  JWARS  do  not  incorporate  concepts  of  operations 
(CONORS)  which  limits  their  utility  for  offense-defense  and  related 
policy  tradeoffs. 

The  number  of  systems,  mission,  and  force-on-force  models 
appear  redundant.  Further,  they  are  largely  decoupled  from  element 
models.  They  are  not  fed  properly  from  lower  levels  nor  are  they 
integrated  between  systems,  mission,  and  force-on-force  levels.  They 
do  not  use  lower  level  models  provided  or  approved  by  the  system 
program  offices  (SPOs). 

As  presently  configured,  M&S  has  little  calibrated  predictive 
capability.  Thus,  the  activity  serves  more  as  an  engineering  adjunct 
than  as  a  full  M&S  program.  The  Task  Force  found  little  evidence  that 
M&S  is  being  guided  by  or  used  to  support  either  BMC  or  systems 
engineering  priority  programs. 
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AND  Simula  tion  Effort 


Findings  (cont.) 

■  MDWAR  system  level  model 

-  Adequate  for  NTB  studies,  training,  &  possibly  system  engineering 

-  Lack  of  comparison  w/  high  fidelity  models  reduces  confidence 

-  Treats  effects  of  potentially  critical  CEC  data  rate  limitations  with 
low  fidelity  lumped  model  approximation 

•  Prevents  analysis  of  CEC  bandwidth  limitation  effects  on  other 
systems  elements 

-  Predictive  capability  limited  by  lack  of  models,  calibration 

-  Has  not  been  provided  to  the  user  as  a  model  of  BMDS  block 
performance 

•  Activity  level  effort  is  primarily  in  C2  and  battle 
management 

-  Incomplete  and  delayed  by  frequent  changes  in  scope  and 
priorities 

-  Work  on  C2/BMC  appears  to  be  in  conflict  with  analyses  needed 
for  confidence  in  performance  of  IDO 


MDWAR  seems  adequate  for  the  NTB  studies  and  training 
simulations  for  which  it  was  designed  and  possibly  for  some  system 
engineering  simulation  use.  Its  flexible  treatment  of  man-in-loop 
issues  is  useful  in  studying  C2  for  global  Integrated  Missile  Defense 
(IMD).  However,  the  lack  of  comparison  of  high  fidelity  element 
models  with  those  in  MDWAR  for  validation  and  accreditation 
reduces  confidence  in  its  results  and  utility. 

MDWAR  cannot  treat  cooperative  engagement  capability  (CEC) 
rates  other  than  via  an  effects  (lumped)  model  and  treats  effects  of 
potentially  critical  CEC  data  rate  limitations  with  low  fidelity  lumped 
model  approximation.  It  operates  on  a  ~1  second  time  step.  Hence,  it 
cannot  treat  tlie  kHz  rates  of  CEC  and  advanced  discrimination  in  a 
fundamental  manner.  Nor  can  it  capture  the  impact  of  CEC  and 
advanced  discrimination  bandwidth  requirements. 

MDWAR  lacks  comparisons  and  calibration  for  its  current  models 
and  lacks  fundamental  models  for  BPI,  forward  observation,  and 
advanced  discrimination,  which  limits  its  predictive  capability. 
However,  with  modest  extension,  it  could  adequately  treat  air- 
breathing  tlireats,  U.S.  missiles,  and  offense-defense  integration. 
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At  the  activity  level,  there  is  significant  effort  only  in  improved 
BMC,  which  is  incomplete,  delayed  by  shifting  and  frequent  changes  in 
scope  and  priorities,  and  has  conflicting  objectives  for  IDO. 
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Scope  OF  the  Modeling 
AND  Simula  jion  Effort 


Recommendations 

■  Modeling  &  simulation  development  and  confident 
utilization  need  to  be  improved 

-  Identify  a  system  level  deliverable  which  captures  the  operational 
effectiveness  of  all  elements  in  a  given  Block  and  deliver  it  to 
USSTRATCOM  to  use  in  support  of  defense  of  the  Regional 
Combatant  Commands  (RCCs) 

-  Provide  a  truly  independent  (and  consistent)  source  of  verification, 
validation  and  (separately)  accreditation  for  all  models  which  feed 
into  the  systems  level  model 

-  Integrate  the  now  separate  C2/BM/C  and  systems  level 
approaches  for  model  development  into  one  model  that  assures  a 
common  architecture  (and  common  M&S  representation) 


M&S  development  and  product  confidence  need  to  be  improved. 
The  first  step  is  to  identify  a  system  level  deliverable  that  captures  the 
operational  effectiveness  of  all  elements  in  a  given  block  and 
provides  it  to  USSTRATCOM  to  use  in  support  of  the  relevant 
regional  combatant  commands  (RCCs).  A  well-documented  version 
of  MDWAR  that  has  carefully  calibrated  models  for  each  of  the 
elements  to  be  delivered  in  each  block  could  serve  this  need. 

Careful  calibration  requires  the  development  and  use  of  a 
consistent  and  truly  independent  source  of  verification,  validation, 
and  accreditation  for  all  models  which  are  integrated  into  the 
systems-level  models. 

To  have  the  maximum  impact,  the  C2/BM/ C  and  systems-level 
approaches  need  to  be  integrated  in  a  common  architecture  and  a 
common  M&S  representation.  These  efforts  are  currently  almost 
completely  separate. 
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Recommendations  (cont.) 

■  Explore  establishing  a  single  systems^ievel  model  as  the 
standard  for  integrated  system  performance  assessment 
and  trade-offs  -  MDWAR  a  likely  choice 

-  Require  PMs  to  provide  models  or  representations  of  their 
elements  and  components 

•  With  standard  interface  definitions,  specifications,  and  controls 

•  Accredited  for  specific,  well  defined  purposes 

-  Sort  out  the  redundancies  In  systems-level  models  in  use  and 
development 

-  Establish  USSTRATCOM  as  a  customer  for  the  modeling  and 
simulation  (representing  the  RCCs)  for  the  block  system 
performance  across  the  envelope 


To  realize  the  full  potential  of  M&S— and  to  provide  the  forward- 
looking  guidance  for  decisions  the  MDA  needs— MD A  should 
specify  a  systems-level  model  as  the  standard  for  integrated  system 
performance  assessment  and  trade-offs  and  enforce  its  use  in  decision 
making.  MDWAR  is  presently  the  most  likely  choice.  The  goal  should 
be  to  produce  fixed  software  builds  to  specify  Emergency  Defensive 
Operations  (EDO),  IDO,  and  IMD  blocks  and  provide  them  to  the 
combatant  commanders  for  assessment  and  training.  The  builds 
should  then  be  updated  with  each  hardware  block  and  modification, 
specifying  the  capabilities  of  the  current  hardware.  The  foundation  of 
this  effort  should  be  the  development  and  use  of  a  standard  set  of 
interface  definitions,  specifications,  and  controls. 

MDA  should  require  in  contracts  that  SPOs  provide  models  or 
representations  of  their  elements  and  components  accredited  for 
specified  purposes,  after  sorting  out  the  redundancy  issue  in  systems- 
level  models  in  use  and  in  development.  It  should  then  provide  those 
models  to  USSTRATCOM  (representing  the  RCCs)  as  a  description  of 
the  block  system  performance  across  its  expected  engagement 
envelope. 
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Recommendations  (cont.) 

■  MDA  should 

-  Assume  responsibility  for  supporting  core  phenomenology  and 
physics  models 

-  Add  credible,  higher  fidelity  representations  of  GBI  discrimination 
and  kill  probability  to  the  GMD  system  model 

-  Incorporate  boost  phase  intercept  models  (and  spaced  based 
elements,  as  they  evolve)  into  the  integrated  systems  simulation 

-  Scope  impact  of  high  bandwidth  CEC  and  advanced 
discrimination  needs  on  simulations  and  predictions  of  system 
performance 


To  assure  that  M&S  can  satisfy  both  its  diagnostic  and  predictive 
functions,  MDA  must  assume  direct  responsibility  for  the  needed 
supporting  core  phenomenology  and  physics  models. 

For  M&S  to  be  useful  in  guiding  advanced  data  fusion  and 
discrimination  efforts,  MDA  must  add  credible,  higher  fidelity 
representations  of  ground-based  intercept  discrimination  and  kill 
probability  to  ground-based  midcourse  defense  (GMD)  system 
models. 

To  guide  advanced  technology  efforts,  MDA  must  incorporate 
boost  phase  intercept  models  (and  spaced-based  elements,  as  they 
evolve)  into  integrated  systems  simulations.  The  lack  of  this 
capability  is  impeding  current  attempts  to  evaluate  the  relative  merits 
of  various  basing  modes,  define  the  initial  capability  levels  desired, 
and  assess  their  synergism  with  current  midcourse  systems. 

In  order  to  prevent  the  fragmentation  of  efforts  at  advanced  fusion 
and  discrimination,  MDA  must  scope  the  impact  of  high-bandwidth 
CEC  and  advanced  discrimination  needs  on  simulations  and 
determine  whether  they  can  be  adequately  modeled  in  current 
systems  simulations  such  as  MDWAR  or  whether  they  will  require 
fundamentally  new  approaches. 
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Chapter  2.  Communication  and  Interface  Standards 


Communications  Challenges  in  BMDS 
Modeling  and  Simulation 

Two  Main  Topical  Areas 

■  Area  1 :  Modeling  of  communications  networks  and  Information  exchange 

-  Incorporating  communications  realism  (finite  bandwidth,  latency,  corruption, 
redundancy,  disruption)  to  establish  system  level  impacts  of  practical  problems 

•  “Facts  of  life”  effects  (weather,  EMI,  network  node  availability) 

•  Inconsistent  or  erroneous  implementation  of  standards 

•  Defense  suppression  (e.g.,  denial  of  service  attacks) 

-  Key  modeling  and  simulation  issue  for  area  1:  The  implications  of  communications 
modeling  on  simulation  architecture  and  run  time 

■  Area  2:  Federating  specialized  sims  into  an  end-to-end  engineering 
simulation 

-  Making  models  of  BMDS  functions  “communicate”  with  each  other  and  operate 
causally  to  represent  the  detection-to-intercept  engagement  timeline 

•  Matching  “output  pins"  on  one  simulation  to  “input  pins"  on  another  simulation 

•  Developing  the  ability  to  derive/validate  timing  &  error  budgets  for  BMDS 
engagement  sequences 

-  Key  modeling  and  simulation  issue  for  area  2:  MDA  simulations  and  simulation 
architectures  capability  to  support  end-to-end  engineering  analyses/verification 


Communications  -  the  physical  transmission  of  information 
between  nodes  in  a  simulation  and  the  more  abstract  relationship 
between  models  &  simulations  in  an  end-to-end  federation 
(simulation  of  simulations)  -  plays  an  important  role  in  the 
engineering  of  the  Ballistic  Missile  Defense  System  and  in  the  reliable 
assessment  of  BMDS  capabilities.  Communications  plays  a  vital  role 
in  at  least  two  ways. 

The  first  is  in  the  timeline  realism  that  is  introduced  by  modeling 
the  system  communications  paths,  attendant  message  flows,  and 
message  service  delays.  Getting  the  message  traffic  flow  right  -  as 
well  as  the  physics-based  and  human-in-the-loop  events  that  trigger 
that  flow  -  is  a  large  step  towards  understanding  the  timeline 
impacts  of  what  is  often  aissumed  away  (or  more  positively,  assumed 
to  be  working)  in  top-level  architecture  performance  assessments. 

The  main  issue  in  this  arena  is  the  ability  to  incorporate 
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communications  modeling  ex  post  facto  without  having  to  pull  the 
host  simulation  apart  and  without  seriously  impacting  run  time. 

The  second  role  of  communications  is  as  the  linkage  between 
models  that  might  make  up  a  federated  end-to-end  system 
engineering  simulation.  Such  an  end-to-end  simulation  would  be 
potentially  valuable  for  engineering  design  and  verification  of  BMDS 
engagement  sequences.  Communications  in  this  context  applies  to 
the  ability  of  element  (including  C2/BM/ C),  component,  and 
auxiliary  models  (such  as,  sensor  energy  management,  data 
association,  fusion,  and  tracking  tools;  common  environmental  and 
signature  codes;  test  bed  implementations  of  discrimination  and 
decision  algorithms)  to  signal  each  other  and  to  act  upon  information 
passed  with  standardized  message  formats  across  standardized 
interfaces. 

Though  some  models  and  simulations  developed  either  at  MDA 
or  at  the  element  program  offices  were  conceived  to  communicate 
with  a  centralized  executive  over  Distributed  Interactive  Simulation 
(DIS)  or  High  Level  Architecture  (HLA)  interfaces,  others  were  not. 
Further,  most  were  not  designed  to  work  with  each  other  or  to  act  on 
messages  that  reflect  a  network-centric  fire-control  approach.  The 
main  issue  in  this  arena  is  the  practicality  of  constructing  engineering 
and  verification  tools  that  leverage  significant  investments  at  MDA  in 
modeling  and  simulation.  Clearly,  an  unattractive  alternative  would 
be  to  architect  and  build  the  required  end-to-end  simulation 
capability  at  MDA  from  the  ground  up. 
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■  There  Is  a  clear  need  for  realistic  communications  modeling  before 
the  workability  of  message-intensive  BMDS  capabilities  can  be  fully 
understood 

-  Understand  bandwidth  constraints  on  performance  of 

•  Project  Hercules  Decision  Architecture 

•  Integrated  Fire  Control  (Launch  of  Remote,  Engage  of  Remote) 

•  Target  Object  Map  Transmission  and  Feature  Aided  Track 
Association 

•  Plot  Fusion  (CEC-Like)  and  Network  Based  Discrimination 

■  Message-by-message  modeling  with  queue  management  and  node- 
by-node  network  modeling  will  throttle  simulations  designed  for  real 
time  or  faster  than  real  time  operation  (e.g.  wargaming  applications) 

-  Given  current  computing  power  at  the  JNIC 

•  Non-real  time  BMD  system-level  engineering  simulations  can  handle 
almost  arbitrary  communications  fidelity  and  should  in  the  interest  of 
grasping  the  difference  between  BMDS  concepts  and  implementation 


Transformational  communications  promises  to  eliminate  network 
throughput  and  message  service  delays  as  BMDS  issues.  In  the 
meantime,  finite  bandwidth  communication  pipes  will  be  both  an 
enabler  of  advanced  BMDS  concepts  and  a  potentially  powerful 
constraint  on  performance.  Faithful  modeling  of  communications  is 
essential  to  understanding  the  difference  between  principles  and  real 
world  practice. 

Message  intensive  concepts  like  integrated  fire  control  and  data 
fusion  are  likely  to  preclude  the  explicit  modeling  of  communications 
traffic  in  real  time  simulations  used  for  CONOPS  testing  and  operator 
training.  On  the  other  hand,  design  and  verification  simulations  can 
and  sJtould  attack  the  transmission  and  handling  of  information  down 
to  the  bit  level  so  that  MD  A  engineers  can  get  a  handle  on  how 
communications— normally  assumed  to  be  what  it  must  be  in 
architecture-level  analyses— modifies  ideal  BMDS  performance. 
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■  Develop  fast  running  effects  models  for  real  time 
simulations  which  still  capture  the  impact  of  finite 
communications  resources  on  BMDS  operational  flexibility 
and  performance 

■  Federate  or  integrate  realistic  communications  models 
with  high  fidelity  system-level  simulations  used  for 
engineering  and  verification  analysis 

-  Federate  where  possible  with  parts  of  existing  codes 

♦  e.g.,  use  EADTB  for  definitive  Link-16  model 

-  Initiate  new  communications  modeling  efforts  to  support  detailed 
analyses  of  the  new  message  intensive  operational  concepts 


War  gamers  on  consoles  at  the  JNIC  are  not  overly  troubled  by  the 
details  of  networks  and  communications.  On  the  other  hand,  they  are 
interested  in  real  time  workloads  and  the  relationship  between  the 
decisions  they  make  and  the  BMDS  behavior  they  get. 

Representation  of  communications  throughput,  network  processing, 
and  message  latency  are  important  to  both  cases.  Real  time 
functioning  of  war  gaming  simulations  will  require  a  fast  running 
"effects  model"  of  the  communication  system  that  distills  the  details 
into  simple  quality  metrics  and  delays  that  still  permit  hona  fide 
operational  problems  to  be  identified.  Engineering  design  and 
verification  simulations  will  require  far  more  detail.  In  this  case,  real 
time  operation  is  not  the  overriding  concern.  Federation  with 
community  standard  communication  models  (e.g..  Extended  Air 
Defense  Test  Bed  [EADTB]  for  Link-16)  should  be  the  first  choice, 
with  new  models  being  built  only  as  necessary. 
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Area  2:  End-to-End  Capabilities  Sought 


■  A  means  to  determine  if  al!  BMDS  functions  between  detection  and 
intercept  can  in  truth  be  executed  in  the  engagement  time  available 

■  A  means  of  identifying  all  execution  bottlenecks  and  failure  modes 
that  are  not  the  result  of  accidents  or  poor  quality  control 

•  An  end-to-end  simulation  capability  that  is  functionally  complete 

-  Captures  all  the  key  factors  in  probability  of  engagement  success  (PES) 
chain 

■  A  federated  architecture  that  maximizes  reuse  of  MDA  core  models 
and  element  simulations  and  that  can  accommodate  new  models  over 
time 

-  Element  models  and  communications  models 

-  BMD  Benchmark  for  tracking  in  multiple  object  environments 

-  Hercules  Test  Bed  for  radar  beam  and  energy  management 

-  Hercules  Fusion  Toolbox 


MDA's  need  for  an  end-to-end  engineering  simulation  capability 
is  closely  analogous  to  computer  chip  design  and  manufacture.  A 
manufacturer  wants  to  develop  a  new  computer  chip  that  will  clock 
at  speed  X.  The  design  team  lays  out  the  connections  and  logic  gates 
and  looks  for  some  way  to  validate  the  logic  before  handing  the 
blueprint  over  to  the  assembly  line  and  committing  to  the  expense  of 
full-scale  production.  Will  the  chip  really  clock  at  speed  X  or  is  that 
just  a  drawing  board  fantasy? 

The  NTS  needs  to  answer  similar  questions  in  the  system 
engineering  job,  preferably  in  the  design  phase  when  mistakes  are 
cheapest  to  fix  and  well  before  surprising  behaviors  are  caught  by 
big-ticket  HWIL  and  live-fire  demonstrations.  The  increasing 
complexity  of  BMDS  interactions  over  time,  and  the  genuine 
potential  for  unpredicted  and  unwelcome  emergent  behaviors, 
dictates  the  requirement  for  an  exploratory  simulation  tool  that  is 
long  overdue  in  its  architecting  and  development.  That  same  tool  will 
also  prove  highly  useful  in  the  verification  phase. 

Emphasis  on  M&S  federation  and  enforcement  of 
communications  standards  recognizes  that  many  of  the  piece-parts  of 
a  useful  end-to-end  system  engineering  capability  already  exist. 
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though  tliey  have  not  been  developed  from  a  master  plan  under 
centi'alized  management.  There  should  be  no  need  to  rebuild  those 
tools.  Likewise,  new  models  must  find  an  easy  way  to  link  into  the 
end-to-end  simulation  framework  as  the  BMDS  evolves. 
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Given  A  Well  Thought  Out  Simulation  Architecture 

■  Common  bed  down 

-  Maintaining  identical  views  of  the  battlefield  between  simulators 

-  All  element  positions  In  one  simulation  must  be  the  same  in  all  other 
simulators 

■  Assigning  executive  authority 

-  Accepting  start  and  stop  commands  from  alien  simulations 

■  Execution  speed 

-  Real  time  to  multiples  (and  sub-multiples)  of  real  time 
•  Time  synchronization 

-  Measurement  and  control  of  time  differences  between  simulators 
«  Environmental  data  exchange 

-  Enforcing  common  threat  and  environment 

-  Ensuring  that  when  one  sIm  affects  the  environment,  all  other  sims 
know  it 

■  Perceived  data  exchange,  Including  tactical  messaging  system 

-  Passing  digital  data  to  and  from  tactical  battlefield  messaging  systems 


Though  conceptually  powerful,  M&S  federation  will  require 
careful  forethought.  Enforcement  of  functional  causality,  time 
synchronization,  common  environments,  messaging  standards,  and 
decision  making  protocols  are  clear  challenges. 
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Area  2:  Findings 

■  Enforcement  of  DIS  and  HLA  interface  standards  means  that  any 
simulation  could  interact  with  another  through  a  master  simulation 
executive  -  in  principle 

•  On  the  other  hand,  most  missile  defense  models  are  not  designed  to 
communicate  except  in  this  most  formal  way  -  bits  flow  both  ways, 
what  about  information? 

~  Models  for  the  most  part  not  functionally  specialized 
Models  not  designed  to  plug  into  some  overall  intent 

■  Element  models  currently  limited  to  tightly  integrated  representations 
of  autonomous  performance 

-  Often,  component  (radar,  C2/BM/C,  interceptor)  representations  cannot 
be  extracted 

-  Messages  may  be  passable  across  an  interface  from  BMDS  C2/BM/C, 
but  wholly  unusable 

■  Move  towards  integrated  battle  management  and  fire  control  (network 
centric  warfare)  will  call  for  component  models  that  can  be  individually 
signaled  and  manipulated 

-  Models  currently  reflect  the  challenge  of  designing  the  actual  BMDS 


Though  DIS  and  HLA  standards  are  not  mutually  compatible, 
simulations  written  to  either  standard  should  be  capable  of  plugging 
into  a  master  simulation  executive.  For  example,  MDWAR  has  the 
capability  to  connect  to  any  other  simulation  that  can  pass  and 
receive  messages  over  a  standard  communications  interface.  The 
ability  to  federate  models  through  a  central  controller  offers  the 
potential  for  development  of  very  powerful  system  engineering  and 
verification  tools  that  make  full  use  of  program  office  investments  in 
high  fidelity  element  representations. 

However,  the  practical  difficulty  of  constructing  end-to-end 
simulations  of  the  BMDS  mirrors  the  difficulties  of  integrating  and 
orchestrating  missile  defense  elements  in  the  real  world.  Models  of 
legacy  elements  tend  to  be  tightly  integrated  representations  of 
autonomous  performance  and  may  be  unresponsive,  by  design,  to 
coordinating  messages  from  C2/BM/ C.  Furthermore,  as  the  BMDS 
moves  towards  more  network  centric  fire  control  implementations 
such  as  launch  on  remote  and  engage  on  remote,  there  will  a  be  a  real 
appetite  for  sensor  and  weapon  models  that  can  be  independently 
signaled  and  manipulated.  Unfortunately,  such  stand-alone  modules 
cannot  be  extracted  from  many  of  today's  element  models  which 
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were  designed  to  simulate  integrated  element  performance  and  not 
the  detailed  behavior  of  individual  components. 
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•  C2/BM/C  and  system  engineering  national  teams  both  see  need  for 
end-to-end  engineering  design/verification  tools  and  the  need  for  a 
federation  of  interacting  models 

-  However,  each  are  taking  their  own  approach  with  little-to-no  crosstalk 

-  Performance  certifications  not  likely  to  be  in  a  common  M&S  frame  of 
reference 

■  No  end-to-end  system  engineering  and  verification  tool  exists  today 

-  Difficult  to  understand  how  BMDS  could  be  put  on  alert  without  such  a 
tool 

-  Also  not  clear  to  everyone  what  the  problem  would  be  if  that  tool  did  not 
exist 

■  BMDS  simulation  design  driven  by  a  combination  of  past  practice, 
service  priorities,  computer  science,  consensus,  and  happenstance 

-  No  engineering  analysis  plan  against  which  M&S  completeness  could  be 
judged 

-  A  framework  of  engagement  sequences  and  event  traces  exists  today 
that  could  drive  modeling  and  simulation  requirements  and  could 
establish  relationships  between  models,  but  it  is  not  being  used  that  way 
yet  to  any  effect 


Both  the  NTS  and  the  NTB  are  pursuing  end-to-end  simulation 
capabilities  for  their  respective  purposes,  but  they  are  not  in 
meaningful  communication  with  each  other.  This  has  resulted  in 
parallel  development  paths  and  potentially  serious  disconnects. 

There  is  presently  no  end-to-end  engineering  design  and 
verification  tool  available,  nor  is  there  a  clear  recognition  of  the  need 
for  an  associated  development  roadmap.  Element  model  autonomy  is 
an  obstacle  to  the  end-to-end  simulation  of  BMDS  capabilities.  One 
could  ask  the  question  "How  far  beyond  its  design  space  could 
element  X  be  pushed  if  interfaced  to  an  integrated  BMDS  fire  control 
and  discrimination  network?"  However,  the  answer  is  presently 
outside  the  ability  of  MDA  system  engineers  to  address  rigorously. 
Beyond  that,  stand-alone  element  models  might  be  expected  to  accept 
network  messages  over  an  interface,  but  might  also  be  expected  to 
"drop  on  the  floor"  all  messages  it  was  not  programmed  to  process. 
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■  Communications  between  models  is  an  ideal  not 
implemented  in  practice 

-  More  than  just  the  enforcement  of  communications  standards 

-  True  modeI-to-mode(  communications  means  the  ability  of  each 
model  to  operate  on  information  from  the  outside,  implement 
decisions,  and  pass  information  back  across  an  Interface 


The  problem  of  federating  models  and  simulations  which  were 
not  designed  to  be  interlocked,  and  which  were  constructed  to  no 
formal  standards  of  fidelity,  is  a  formidable  challenge,  though  one 
worth  taking  on. 
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•  Develop  functionally  complete,  end-to-end  system  engineering  tools 
for  detailed  evaluation  of  event  trace  timing,  validity,  and  error 
budgets 

■  Investigate  move  to  a  federated  simulation  architecture  for  high  fidelity 
end-to-end  engineering  analysis 

-  Maximize  use  of  sunk  MDA  investments  in  specialized  M&S 

•  BMD  Benchmark,  Hercules  Test  Bed,  BM/C3  Element  Support  Task 
(BEST),  MDWAR,  EADTB,  element  and  component  models 

-  Accommodate  upgrades  without  major  recoding 

■  Add  interoperability  requirements  to  all  existing  system  engineering 
contracts  for  development  of  BMD  component-level  models  and 
include  similar  language  in  all  future  contracts 

-  Define  and  enforce  model  communications  standards 

•  Move  towards  network  centric  warfare  will  drive  emphasis  towards 
signaling  and  manipulation  of  components  (launchers,  radars),  not 
just  elements 


The  recommendations  are  straightforward: 

1 .  Develop  an  end-to-end  system  engineering  tool  for 

validation  of  BMDS  engagement  sequences  and  event  traces. 
Such  a  tool  is  essential  for  understanding  and  allocating 
timing  and  error  budgets  end-to-end  and  for  understanding, 
during  the  BMDS  design  phase,  that  concepts  will  really 
work  as  advertised. 

2  .  So  as  to  maximize  the  sunk  cost  investment  in  M&S  at  MDA, 

a  federated  approach  should  be  investigated. 

3  .  All  future  M&S  development  should  be  initiated  with 

interoperability  in  mind  to  keep  pace  with  evolving 
architectural  concepts  that  stress  true  integration  rather  than 
the  orchestration  of  elements  originally  designed  for  stand¬ 
alone  operation. 
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Chapters.  Fidelity 


Fidelity  Needs  and  Issues 


■  The  classes  of  MDA  M&S  activities  examined  range  from 
phenomena  to  campaign  -  from  constructive  to  live 

■  “Appropriate  fidelity”  is  the  minimum  fidelity  needed  to 
answer  the  question  being  addressed 

■  The  detail  of  parameter  inputs  should  match  the  fidelity 
and  application  of  model  outputs 

-  Detailed  models  can  include  hundreds  of  parameters 

-  Parameters  that  drive  models  can  be  difficult  to  identify 

-  Unknown  sensitivity  of  simulation  outputs  to  these  parameters 
exists 

-  A  poorly  estimated  parameter  makes  it  difficult  to  determine  that 
the  simulated  results  may  be  questionable 


Global  Fidelity  Concerns  by  M&S  Class 


•  Threat,  engineering,  phenomenology,  and  lethality  models 

-  e.g.,  Strategic  Scene  Generation  Model  (SSGM),  BEST,  Parametric 
Endo-Ex-lethality  Simulation  (PEELS),  PEGEM.... 

-  Detailed  simulations  often  founded  upon  uncertain  or  unavailable  data 

-  Nuclear  environment  not  significantly  addressed 

■  Element  level  models 

-  e.g.,  THAAD,  PAC3,  GBM,  Aegis.  ABL,  KE.... 

-  Class  most  likely  to  benefit  from  improved  HWIL  activity 

-  Element  contractors  a  key  source  of  M&S  data  and  resources 

•  Level  of  Involvement  and  coordination  inadequate 

-  BMDS  HWIL  evaluation  models  need  to  be  well  anchored  via  flight  test 

■  Integrated  system  level  MDA  models 

-  e.g..  EADSIM,  EADTB,  MDSE,  MDWAR.... 

-  Primarily  computer/software  and/or  operator  in  the  loop  based 

-  Currently  focused  on  demonstrating  connectivity  -  need  to  also  be 
focused  on  developing  architecture  and  systems 

-  Element  program  involvement  often  uncoordinated 
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Observations  with  respect  to  M&S  fidelity  have  been  segregated 
into  three  general  classes:  phenomena  modeling,  element  modeling, 
and  integrated  element  or  core  modeling. 

SSGM,  BEST,  PEELS,  and  PEGEM  are  representative  of  the 
phenomena  class  of  M&S.  This  class  encompasses  underlying  threat 
characteristics,  environment,  and  lethality  models  used  to  develop 
the  BMDS.  The  primary  overall  concern  with  respect  to  this  class  is 
the  appropriateness,  value,  and  utility  of  the  underlying  database  of 
measurements  used  to  anchor  the  models.  Additionally  no  clear 
position  has  been  taken  with  respect  to  including  a  nuclear 
environment  into  the  BMDS. 

THAAD,  PAC-3,  GMD,  ABL,  and  Kinetic  Interceptors  (KI) 
modeling  activities  are  representative  of  the  element  class  of  M&S. 
The  primary  concern  with  respect  to  this  class  of  M&S  is  the  role  and 
the  level  of  hardware  inclusiveness  that  HWIL  simulators  are  playing 
in  the  BMDS  development.  Additionally,  roles  and  responsibilities 
among  the  various  participants  in  this  area  are  not  always  clear. 

EADSIM,  EADTB,  MDSE,  and  MDWAR  are  representative  of  the 
integrated  element  or  core  class  of  M&S.  The  relative  importance  and 
the  roles  of  commander-in-the-loop,  operator-in-the-loop,  and 
hardware-in-the-loop  activities  and  how  these  activities  are  balanced 
is  an  area  of  concern.  Additionally,  the  purpose  for  which  these  M&S 
activities  have  been  developed  and  how  they  are  employed  needs  to 
be  balanced  between  competing  evaluation  and  architectural  design 
objectives. 
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Some  Continuing  Systemic  M&S  Fidelity 
and  Associated  W&A  Challenges 

■  Technical 

-  Demands  of  hard-to-kill  (HTK)  end  game  require  a  higher  level  of  fidelity 
in  the  ground  simulations  (both  digital  and  hardware-in-the-loop  (HWIL)) 
(Independent  Review  Team  (IRT)  98) 

■  Programmatic 

-  Accelerate  the  upgrade  of  a  HWIL  facility  adequate  for  credible  testing  of 
the  EKV  in  a  variety  of  endgame  geometries  against  a  variety  of  threats 
(IRT  99) 

-  Vital  information  needed  for  development  and  decisions  will  come  from 
simulations  /  HWIL  anchored  with  flight  test  data  (IRT  1999) 

-  VV&A  as  presently  practiced  with  respect  to  M&S  techniques  is  not 
sufficiently  disciplined  to  inspire  confidence  in  their  use  in  T&E  (DSB 
1999) 

-  The  task  force  determined  that  the  majority  of  problems  associated  with 
DoD  software  development  are  a  result  of  undisciplined  execution  (DSB 
2000a) 

■  Fiscal 

-  More  investment  needed  in  conceptual  modeling,  phenomenology,  and 
experimentation  to  gather  realistic  input  data  in  order  to  provide  decision 
makers  with  confidence  in  M&S  results  (DSB  1999) 


Many  of  the  current  fidelity  concerns  have  been  observed  and 
identified  during  prior  M&S  program  reviews  and  remain  relevant 
today.  In  part  it  was  the  existence  of  these  systemic  issues  that 
became  the  rational  for  establishing  a  coordinated  M&S  program. 
Such  legacy  factors  continue  to  limit  the  M&S  program  in  achieving 
its  full  potential  and  value. 
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■  Miss  distance  was  traditionally  the  standard  metric  for  MDA  M&S 
fidelity 

■  HTK  as  a  concept  has  been  verified  by  flight  experiment 

-  There  will  never  be  sufficient  resources  for  flight  testing  alone  to  provide 
confidence  in  the  BMDS  across  the  full  envelope  of  operations 

-  M&S  at  any  level  will  use  less  resources  than  live  flight  testing 

-  M&S  is  the  only  viable  means  to  explore  additional  HTK  capability  space 

■  Conventional  wisdom 

-  HTK  thought  to  require  high  fidelity  modeling  of  every  nuance  in  order  to 
faithfully  evaluate  miss  distance 

■"  Less  often  appreciated 

-  An  interceptor  sensitive  to  every  nuance  is  poorly  designed 

-  The  impact  of  uncertainty  and  the  buildup  of  consequences 

-  The  threat  will  never  be  understood  to  a  comparable  degree  of  accuracy 

■  The  most  important  factor  in  HTK  capability  evaluation  at  the  intercept 
envelope  edges  will  be  HWIL  testing  of  acceptable  fidelity 


The  traditional  metric  for  missile  defense  has  been  its  ability  to 
faithfully  model  "hit-to-kill"  (HTK)  engagements.  This  metric  can 
serve  as  a  basis  for  defining  the  issues  as  well  as  for  providing 
motivation  when  addressing  missile  defense  M&S  fidelity.  Although 
HTK  has  been  verified  as  a  concept,  the  expense  associated  with 
flight  tests  are  such  that  there  will  never  be  sufficient  resources  to 
either  obtain  quality  statistical  data  about  any  single  flight  construct 
or  to  address  performance  at  the  edge  of  the  capability  envelope. 

Both  an  understanding  of  the  threat  and  of  interceptor 
performance  is  critical  for  a  successful  HTK.  Yet,  in  the  real  world, 
the  threat  will  never  be  understood  to  the  degree  preferred.  A  HTK 
paradox  is  that  a  well-understood  threat,  or  assuming  a  well- 
understood  threat,  can  result  in  a  fragile  design  whereas  a  poorly 
understood  threat  can  result  in  a  robust  design. 

The  principal  mechanism  available  to  understand  overall  missile 
defense  performance,  within  realistic  economic  constraints,  is 
through  the  use  of  "hardware-in-the-loop"  testing. 
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Hardware-in-the-Loop  Simulation 


■  Current  Kill  Vehicle  (KV)  HWIL  fidelity  demand  concentrates  on  the 
seeker 

-  HWIL  seeker  with  synthetic  imaging 

-  Other  factors:  HWIL  needs  to  address  missile  body,  IMU,  controls,  valves, 
etc. 

-  Subtle  integration  errors  potentially  missed 

■  Expensive  to  obtain  high  fidelity  in  an  integrated  system  but  cost  is 
still  lower  than  flight  testing 

-  Test  (imitations  may  make  flight  testing  less  valid  at  the  system  level  than 
the  right  level  of  HWIL  testing 

■  Current  HWIL  testing  tends  to  involve  hardware  that  is  either 
convenient  or  pushing  the  state-oRhe-art  rather  than  what  is  required 
or  appropriate 

■  Relative  roles  and  responsibilities  with  respect  to  Element  Programs, 
Service’s  and  MDA  were  not  clearly  defined 


While  HWIL  testing  can  be  expensive,  its  costs  are  likely  to  be 
modest  compared  to  flight  testing.  The  Task  Force  found  that  some 
tested  HWIL  assemblies  are  not  as  inclusive  as  they  might  be  with 
respect  to  the  actual  flight  hardware. 

Most  HWIL  interceptor  simulations  are  focused  on  evaluation  of 
the  seeker,  associated  processors,  and  related  algorithms.  This  is  not 
surprising  as  the  seeker  is  a  critical  component.  Equally  important 
however  is  the  remainder  of  the  interceptor  including  its  more  basic 
components  such  as  inertial  measurement  unit  (IMU),  controls, 
valves,  and  functional  responses  such  as  missile  body  flexure 
dynamics.  Typically  tiiese  other  factors,  even  when  included,  are  not 
fully  incorporated  into  the  HWIL. 

Some  hardware  often  is  only  bench  mounted  and  not  subjected  to 
the  dynamic  environment  necessary  to  be  realistically  exercised  and, 
as  a  result,  subtle  integration  errors  can  be  missed.  There  have  been 
exceptions  to  this  HWIL  testing  observation,  such  as  hover  tests,  or 
combined  synthetic  target  generation  full  up  dynamic  table  tests  of 
more  complete  versions  of  the  test  article.  However,  such  higher 
quality  HWIL  testing  remains  the  exception  rather  than  the  rule.  At 
the  element  level  the  critical  source  of  HWIL  data,  hardware,  and 
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testing  capability  resides  with  the  major  defense  acquisition 
programs  (MDAFs).  However  the  relative  roles  and  responsibilities 
between  the  MDAPs,  the  Seiwices,  and  MDA  remain  unclear. 
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The  Impact  of  Uncertainty 


■  Uncertainty  in  M&S  fidelity  is  the  difference  between  “idealized  behavior”  and 
reality  considering  the  element  model,  the  threat,  and  the  required 
performance 

■  Uncertainty  in  fidelity  is  addressed  through  sensitivity  analysis  across  the  full 
range  of  the  uncertainty 

-  Identifies  what  has  to  be  modeled  at  what  level  of  fidelity 
“  Earlier  DSB  report  recommendations  remain  relevant 

-  Traditionally  what  is  well  understood  (i.e.,  the  environment)  is  modeled  at  high 
levels  of  fidelity  regardless  of  the  sensitivity  of  the  result  to  the  phenomena 

-  Conversely  what  is  not  well  understood  is  modeled  simplistically  regardless  of  its 
importance 

■  BMDS  system  design 

-  Assuming  a  well  defined  threat  can  lead  to  a  brittle  system  design 

-  Recognizing  the  inherent  uncertainty  leads  to  a  robust  system  design 

■  Significant  MD  threat  uncertainty  exists  due  to 

-  Lack  of  intelligence  data 

-  Manufacturing  realities 

-  Threat  responses 


Uncertainty,  that  is  the  difference  between  reality  and  the  model, 
plays  a  critical  role  in  understanding  M&S  fidelity.  One  objective  of 
sensitivity  analysis  is  to  ensure  that  those  parameters  that  have  the 
most  impact  on  the  simulation  performance  are  accounted  for  during 
model  development  in  proportion  to  their  importance.  An  earlier 
DSB  report  found  that  often  what  is  well  understood  (for  example  the 
environment)  is  modeled  more  extensively  (consequently  employing 
more  resources)  than  that  which  is  not  well  understood,  regardless  of 
its  relative  importance  to  overall  simulation  performance.  This  is  an 
example  of  parameter  uncertainty  and  sensitivity  analysis  being 
improperly  applied  to  missile  defense  M&S. 

Of  singular  importance  to  the  current  M&S  fidelity  issue  is  the 
observation  that  there  will  always  be  uncertainty  associated  with  the 
threat,  either  because  of  a  lack  of  intelligence  data,  because  of 
manufacturing  uncertainties,  or  because  of  deliberate  uncertainties 
introduced  by  the  potential  enemy.  This  reality  establishes  a  limit  to 
the  value  of  certain  modeling  parameters. 
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Threat  Modeling 

•  BEST  is  the  principal  high  fidelity  M&S  activity  associated  with  threat  signature 
development 

-  Brings  code  languages  up  to  date 

-  Replicates  existing  physics  code  (OSC.  XPATCH,  etc.) 

•  The  level  of  fidelity  has  probably  reached  the  point  of  diminishing  returns 

-  Further  fidelity  of  threat  signature  modeling  unlikely  to  benefit  BMDS 

•  If  assumed  representative  may  be  counterproductive 

•  An  example  is  early  conclusions  reached  about  midcourse  discrimination  capability  via 
EO/IR  observations 

■  Lethality  modeling  fidelity  suffers  from  a  similar  deficiency 

-  An  exception  is  hypervelocity  phenomena  -  unavailable  via  ground  test 

•  Sensitivity  to  input  assumptions  is  a  critical  modeling  result 


Performance 

metric 

(ie.,  Hii.v.v  distance,  etc.} 


c=^>• 


Threat  Vneertainty  Space 


BEST  is  the  acronym  for  current  M&S  activity  associated  with 
phenomena  development  involving  the  electro-optic  (EO)  and  radio 
frequency  (RF)  characteristics  of  the  environment  and  threat 
signature.  BEST  brings  existing  code  such  as  OSC  and  XPATCH  up 
to  date  and  incorporates  this  code  into  a  common  framework.  With 
respect  to  midcourse,  at  least,  this  activity  has  probably  reached  the 
point  of  diminishing  returns.  That  is,  the  level  of  modeling  detail  has 
reached  that  point  where  it  exceeds  the  expected  threat  uncertainty. 

This  situation  is  illustrated  in  the  accompanying  figure  which 
represents  how  the  threat  characteristics  are  mapped  by  the  element 
model  to  the  desired  response.  Significant  activity  has  been 
expended  in  characterizing  the  threat  to  a  level  of  detail  that  will 
provide  the  desired  performance;  for  example,  an  acceptable  miss 
distance.  However  the  uncertainty  associated  with  the  threat 
generally  exceeds  the  actual  modeled  detail  signature  of  the  threat. 
The  most  robust  missile  defense  design  is  one  that  will  provide 
acceptable  performance,  given  the  more  realistic  uncertainties 
expected.  Modeling  the  system  with  tighter  threat  parameters  can 
produce  overconfidence  and  may  be  counterproductive. 
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A  similar,  but  less  serious,  situation  exists  in  the  lethality  area 
where  uncertainty  in  the  threat  becomes  a  basis  for  modeling  and  for 
drawing  conclusions  about  performance.  However,  in  this  case,  there 
is  little  reliable  data  on  hypervelocity  impacts  and  the  associated 
models  are  poorly  anchored  in  that  regard.  Activity  to  anchor  such 
models  should  continue  as,  and  if,  such  data  becomes  available. 
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BMDS  System  Level  Modeling 

■  Fidelity  at  any  particular  level  of  the  BMDS  hierarchy  will  depend  upon 
the  problem  to  be  addressed  at  each  level,  not  on  the  level  itself 

■  Higher  level  (i.e.,  cookie  cutter,  etc.)  solutions  generally  assume  the 
answer  will  be  decoupled  from  other  program  inputs 

-  Not  clear  decoupling  assumptions  are  valid  in  current  models 

■  Focus  of  higher  level  modeling  fidelity  has  been  on  interoperability, 
communications,  and  CONORS  development 

■  There  appears  to  be  no  current  M&S  too!  of  appropriate  fidelity  for 
future  block  architecture  development 

-  Of  equal  importance  to  current  objectives 

-  Needs  to  be  considered  and  designed  into  developing  models 

-  Unclear  that  NTB  and  NTS  M&S  objectives  are  in  agreement 


It  is  often  implied  that  M&S  fidelity  increases  as  model  detail  is 
added  to  those  simulation  levels  closer  to  the  individual  hardware 
components  and  decreases  as  modeling  is  aggregated  to  higher 
levels.  While  the  accuracy  may  increase  with  model  detail,  fidelity 
might  not.  Aggregation  to  higher  missile  defense  levels  takes  many 
forms  and  the  actual  level  of  fidelity  achieved  will  depend  on  the 
question  being  asked  and  the  purpose  for  which  the  model  was 
developed. 

Higher  level  system  simulations  generally  assume  that  the  answer 
will  be  decoupled  from  other  program  inputs,  yet  it  is  not  clear  that 
decoupling  is  valid  in  current  models.  These  effects  are  discovered 
only  by  applying  end-to-end  engagement  simulations,  where 
statistical  performance  is  assessed  by  Monte  Carlo  iterations  of  the 
entire  set  of  coupled  processes. 

Examining  the  higher  level,  or  core  missile  defense  models,  it  is 
clear  that  the  primary  program  focus  in  this  area  has  been  on  the 
interoperability,  communications,  and  CONOPS  functions.  Although 
several  models  were  available  that  could  contribute  to  developing 
and  evaluating  future  block  architectures,  no  higher  level  M&S  tool 
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was  found  that  was  available  to  uniquely  perform  this  critical 
function. 
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Flight  Testing  and  Model  Anchoring 

■  Flight  test  flexibility  will  always  be  limited 

-  Cost  and  schedule  constraints 

-  Need  to  demonstrate  progress 

-  Safety  and  range  capabilities 

■  Objectives  of  flight  testing 

-  To  provide  proof  of  concept 

-  To  support  the  acquisition  process 

-  To  verify  the  BMDS  model 

■  The  objective  determines  those  flight  constructs  selected  and  fidelity 
of  the  model  anchoring 

-  Examining  system  capabilities  at  the. edge  of  the  performance  envelope 
requires  extending  flight  test  data  into  new  regimes,  with  associated  risks 

-  Interpolation  vs  extrapolation  will  be  a  factor  In  construct  selection 

■  Focus  should  be  on  operational  system  risk  management  not  flight 
test  risk  management 

-  If  flight  testing  is  dominated  by  need  for  success,  then  we  will  not  find  out 
what  we  don’t  know 

-  Structure  flight  test  program  to  support  BMDS  modeling  objectives 


Linking  simulations  and  models  to  real  world  performance  is 
critical  for  fidelity.  The  ultimate  linkage  is  by  way  of  a  full  up  flight 
test  program  to  anchor  the  BMDS  system  model.  Such  testing  is  a 
complicated  process  that  will  always  be  limited  by  a  variety  of 
constraints.  An  important  question  to  address  is  the  objective  of  such 
a  flight  test  program.  One  purpose  of  a  flight  test  program  is  to 
provide  a  means  to  manage  operational  system  risk. 

Since  there  will  never  be  sufficient  flight  tests  to  fully  characterize 
the  BMDS  within  its  operational  space,  and  certainly  not  at  the 
performance  envelope  edges,  flight  tests  should  be  constructed  with 
emphasis  on  model  verification  as  a  goal,  as  well  as  to  verify 
operational  issues.  Focusing  on  BMDS  model  verification  and 
selecting  flight  test  constructs  with  this  as  one  objective  can  increase 
risk.  However,  data  resulting  from  a  broader  test  spectrum  will 
enhance  confidence  in  the  M&S  program  to  predict  performance  for 
the  BMDS  across  a  broader  spectrum  of  capabilities. 
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■  The  fidelity  of  the  threat  signature  models  for  midcourse 
are  probably  at  the  point  of  diminishing  returns  with  the 
exception  of: 

-  Lethality  anchoring  data  (e.g.,  hypervelocity  KKV  data) 

-  ‘‘Advanced  Discrimination”  and  ‘‘Forward  Based  Sensing” 
approaches  that  do  not  rely  on  detailed  “a  priori”  data 

■  Element  HWIL  models  with  appropriate  fidelity  important 
to  explore  capabilities  not  amenable  to  flight  test 

-  HWIL  can  and  should  be  exercised  at  a  higher  level  of  integration 
than  just  the  seeker  and  associated  algorithms 

-  Consider  structuring  flight  tests  to  also  support  model  validation 
into  areas  of  new  capability 

-  Manage  operational  system  risk  rather  than  flight  test  risk 


There  are  three  primary  conclusions  diat  can  be  drawn  from  the 
previous  analysis  of  fidelity,  one  for  each  class  of  M&S:  phenomena, 
element  level,  and  core  or  integrated  element  modeling. 

With  the  possible  exception  of  hypervelocity  lethality,  threat 
signature,  and  phenomena  modeling  of  exoatmospheric 
engagements,  the  fidelity  of  the  threat  signature  models  for 
midcourse  is  close  to  the  point  of  diminishing  returns  and  resources 
might  be  adjusted  accordingly. 

Increasing  the  level  of  HWIL  M&S  activity  along  with  an  increase 
in  the  associated  degree  of  hardware  involvement  for  the  element 
level  programs  is  important  to  explore  system  capabilities  at  the 
envelope  edges.  The  flight  test  program  should  be  structured  with 
model  verification  as  one  primary  objective,  in  addition  to  its  odier 
program  objectives. 

Integrated  element  or  core  model  development  should  address 
architecture  and  block  change  development  as  well  as 
interconnectivity.  There  is  no  dedicated  BMDS  architecture  model 
capable  of  evaluating  block  change  alternatives,  including  a  common 
set  of  metrics  for  such  decision  making  purposes. 
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Recommendations  -  Threat  Phenomena  and 
Environmental  Fidelity 

■  Threat  phenomena  modeling  fidelity 

-  Consider  limiting  further  expenditure  of  resources  in  those  areas 
of  M&S  associated  with  detailed  payload  threat  signature 
development  for  midcourse 

-  Conserve  related  resources  associated  with  the  gathering, 
exploitation,  testing,  measurement,  and  verification  of  such 
detailed  payload  threat  signatures 

-  Continue  to  obtain  data  for  M&S  purposes  via  intelligence  and 
surrogate  means  as  related  to  the  threat  and  associated  delivery 
capabilities 

-  Exploit  prior  signature  development  programs  by  fully  reducing 
already  available  data  (e.g.,  MSX) 

■  Environmental  modeling  fidelity 

-  Model  environmental  details  at  a  level  of  fidelity  consistent  with 
their  expected  marginal  contribution  to  the  BMDS  effectiveness 

-  Establish  a  corporate  position  with  respect  to  including  nuclear 
effects  in  the  BMDS  modeling  and  simulation 


With  respect  to  the  threat  and  phenomena  areas,  a  primary 
recommendation  is  that  MDA  consider  limiting  additional  resource 
expenditures  associated  with  highly  detailed  threat  signature 
development  normally  associated  with  midcourse.  Data  associated 
with  advanced  discrimination  techniques  which  are  not  "a  priori" 
dependent  should  continue  to  be  obtained.  Such  a  recommendation 
would  also  be  applicable  to  those  supporting  functions  associated 
with  the  gathering,  measurement,  and  verification  of  such  detailed 
exoatmospheric  data.  Threat  capability  data  should  continue  to  be 
obtained,  modeled,  and  existing  data  sources  such  as  Midcourse 
Space  Experiment  (MSX)  should  be  exploited. 

MDA  should  establish  a  corporate  position  with  respect  to  the 
inclusion  of  nuclear  effects  globally  into  the  BMDS  program.  The 
importance  of  sensitivity  analysis  and  its  role  with  respect  to 
environmental  phenomena  code  development  needs  to  be 
emphasized. 
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Recommendations  -  Lethality  Fidelity 

■  The  fidelity  of  PEELS  has  probably  reached  the  point  of 
diminishing  returns 

-  Consider  limiting  further  resource  expenditures  for  lethality  M&S 
activities  which  are  highly  dependent  upon  threat  payload  design 
details 

-  Utilize  hypervelocity  flight  test  data,  when  it  becomes  available,  to 
anchor  exoatmospheric  lethality  models 

-  Continue  M&S  development  activities  with  respect  to  the  impact  of 
the  threat  intercept/detonation  effects 


Also,  consider  limiting  additional  resources  associated  with 
lethality  modeling  effects  that  may  be  unrealistically  dependent  upon 
threat  design  details,  with  the  exception  of  anchoring  hypervelocity 
impact  models  as,  and  if,  relevant  data  becomes  available. 
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Recommendations  -  Element  Fidelity 

■  Improve  the  general  level  of  fidelity  for  “hardware-in-the-loop”  testing 
of  all  BMDS  elements 

-  Expand  the  role  of  HWIL  within  the  BMDS  development  and  its  elements 

-  Incorporate  feasible  hardware  components  and  operational  software  into 
HWIL  simulations 

■  Harmonize  flight  test  activities,  exercises,  wargames,  and  M&S 
program  objectives  to  enhance  confidence  in  BMDS  model  fidelity 

-  Jointly  develop  flight  test  constructs  that  support  both  the  demonstration 
of  the  BMDS  effectiveness  as  well  as  the  extrapolation  of  its  performance 
via  M&S 

-  Consider  developing  flight  test  scenarios  based  upon  their  value  in 
calibrating  the  M&S  toolset  at  the  design  envelope  edges 

-  Expand  flight  performance  statistical  database  and  examine  higher  risk 
“off  nominal"  operations  using  enhanced  HWIL  capabilities 


With  respect  to  fidelity  at  the  element  level,  the  general  role  of 
HWIL  and  its  associated  degree  of  hardware  inclusion  should  be 
elevated  to  provide  for  greater  confidence  in  the  corresponding 
statistical  database  generated  as  well  as  for  performance  excursions 
into  flight  regions  not  addressable  in  the  flight  test  program.  HWIL 
components  should  be  as  close  to  the  flight  configuration  as  possible, 
including  realistic  dynamics  as  applicable.  Most  current  HWIL 
simulations  are  primarily  centered  on  the  seeker  and  its  associated 
algoritlims. 

Developing  flight  test  constructs  should  be  a  joint  activity 
between  the  element  program  office,  test  and  evaluation,  and  the 
M«&S  program  office.  M&S  should  not  be  a  passive  participant  in  the 
flight  test  program  but  should  have  an  active  role  in  the  development 
of  flight  test  scenarios,  structured  specifically  to  enhance  the  fidelity 
and  ability  of  the  BMDS  simulations  to  extend  into  unexplored 
regions  with  confidence. 
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Chapter  4.  Ensuring  the  Validity  of  Simulations 


Selected  Definitions 


■  INDEPENDENT  -  A  knowledgeable  and  experienced  capability  not 
involved  in  the  design  or  development  of  a  model 

■  VERIFICATION  -  The  process  of  determining  that  a  model 
implementation  accurately  represents  the  developer’s  conceptual 
description  and  specifications.  Verifications  also  evaluates  the  extent 
to  which  the  model  or  simulation  has  been  developed  using  sound 
and  established  software  engineering  techniques. 

■  VALIDATION  -  The  process  of  determining  the  degree  to  which  a 
model  is  an  accurate  representation  of  the  real-world  from  the 
perspective  of  the  intended  uses  of  the  model. 

■  ACCREDITATION  -  The  official  determination  that  a  model  or 
simulation  is  acceptable  for  use  for  a  specific  purpose. 

■  BASIS  OF  CONFIDENCE  (BOC)  Document  -  A  document  that 
identifies  the  functionality  and  limitations  of  a  model  and  provides  a 
summary  and  reference  of  W&A  activities.  The  BOC  is  typically  used 
to  support  accreditation  or  use  decisions. 


As  noted  on  this  chart,  independent  verification  is  the  process  of 
determining  that  a  model  implementation  accurately  represents  the 
developer's  conceptual  description  and  specifications.  Verification 
also  evaluates  the  extent  to  which  the  model  or  simulation  has  been 
developed  using  sound  and  established  software  engineering 
techniques.  Validation  refers  to  the  process  of  determining  the 
degree  to  which  a  model  is  an  accurate  representation  of  the  real 
world  from  the  perspective  of  the  intended  uses  of  the  model. 
Accreditation  is  the  official  certification  that  the  model  or  simulation 
is  acceptable  for  use  for  a  specific  purpose.  The  Basis  of  Confidence 
documents  are  used  to  record  the  functionality  and  limitations  of 
models  and  will  be  discussed  later  in  this  section. 
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Key  Independent  VV&A  Needs 

■  Area  1  -  General  scope  and  content  quality  of  VV&A 

~  A  VV&A  effort  across  element,  program,  and  core  models  that 
adequately  tests  the  model  “physics/algorithms”  for  intended  uses 
and  over  "envelope  corners”  to  ensure  that  M&S  predictions  when 
needed  for  blocks,  spirals  and  phases  of  BMD  capability  are  valid 

■  Area  2  -  Process  and  scheduling  rigor  of  VV&A 

-  A  VV&A  process  that  uses  “best  practices”  learned  throughout 
DoD  and  elsewhere 

-  An  adequate  schedule  definition  and  discipline  in  VV&A  to  tie  it  to 
the  complex  blocks,  spirals  and  phases  design/deployment 
schedule 

-  A  talent  and  funding  level  for  W&A  adequate  to  meet  the  above 
needs 


The  DSB  examination  addressed  two  inter-related  fundamentals 
of  VV&A.  The  first  is  ensuring  adequacy  and  completeness  of  the 
technical  aspects  of  the  models.  Are  the  underlying  physics, 
algorithms,  and  engineering  representations  adequately  checked? 
The  second  area  is  the  maintenance  of  uniform  process  and 
documentation.  Does  W&A  employ  best  practices,  have  adequate 
definition  and  discipline,  and  involve  acceptance  by  those 
responsible  for  the  performance  of  the  elements  and  systems  being 
modeled?  Is  the  process  adequately  synchronized  to  the  blocks, 
spirals,  and  phases  of  the  MD A  development  and  deployment 
schedule?  For  both  of  these  basic  areas,  a  corollary  consideration 
relates  to  levels  of  talent  and  funding  available  to  implement  the 
W&A  process.  Another  corollary  involves  "buy-in"  to  the  W&A 
process  by  all  element  program  managers.  Service  operators,  war 
gamers,  and  other  stakeholders. 

During  the  study,  much  W&A  information  from  MDA  was 
received  in  the  (archived)  presentations  to  plenary  sessions.  One 
additional  splinter  session  was  held  on  May  19, 2003,  to  specifically 
discuss  W&A  activity  with  MDA  M&S  managers.  This  session  was 
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particularly  beneficial  for  examining  and  discussing  some  of  the 
documentation  which  time  would  not  have  otherwise  permitted. 

The  overall  summary  observation  reported  to  the  full  Task  Force 
following  this  splinter  meeting,  and  reflected  in  what  follows  in  this 
section,  was  that  further  confidence  was  gained  in  the  existence  of  an 
aggressive  MDA  process  to  establish  and  maintain  VV&A  rigor,  to 
include  filling  in  past  gaps.  The  hard,  detailed  effort  to  fully 
synchronize  VV&A  to  the  various  blocks,  spirals,  and  phases  remains 
to  be  completed. 
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yV&A  Repository 

■  Need:  A  comprehensive  web-based  VV&A  repository  with 
secure  access  to  W&A  documentation  and  track  VV&A 
activities 

■  Recommended  approach:  improve  current  system  with 
collaboration  tools 

•  Current  status 

-  Hosted  at  the  JNIC  /  IDC 

-  Researched  site  enhancement  requirements  and  identified 
possible  software  solutions 

-  Evaluating  alternative  hosting  sites 

■  Issues 

-  Information  assurance,  security,  and  need  to  know 

-  JNIC  /  IDC  need  to  accelerate  updating  site 


This  chart  summarizes  an  approach  to  establish  a  comprehensive 
and  accessible  archive  for  all  model  W&A  documentation.  The  Task 
Force  strongly  favors  this  effort. 
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Findings 


■  The  credibility  of  models  is  mixed 

-  There  is  no  consistent  verification,  validation  and  accreditation 
process  across  MDA  M&S  development 

-  The  Aegis,  PAC-3,  and  THAAD  models  are  generally  carefully 
validated  for  the  purposes  for  which  they  are  accredited 

-  The  GMD  model  treats  some  critical  parameters  such  as 
discrimination  and  overall  probability  of  kill  too  simplistically  for 
adequate  credibility 

•  Sensitivity  to  uncertain  parameters  remains  a  key  issue 
requiring  systematic  treatment  and  reporting 

-  The  ABL  model  departs  from  the  use  of  variable  fidelity 
representations  to  federate  the  ABL  battle  manager  directly 

-  There  Is  no  usable  mode!  for  boost  phase  intercept  systems 

-  No  useable  models  for  space  based  kinetic  interceptors  or 
sensors  nor  space  based  DEW  elements  were  Identified 


PAC-3,  THAAD,  GMD,  Aegis,  Airborne  Laser  (ABL)  models  seem 
adequate  for  their  intended  uses,  for  which  they  have  been  carefully 
accredited,  but  they  are  decoupled  from  each  other  and  from  the 
system  &  activity  layers  of  the  M&S  pyramid.  GMD  model 
treatments  of  discrimination  and  kill  probability  are  oversimplified 
and  do  not  provide  confidence  in  sensitivity  assessments.  The  ABL 
model  just  federates  its  battle  manager.  There  are  no  useful  models 
for  boost  phase  kinetic  or  directed  energy  intercept  from  the  surface 
or  space. 

Current  background,  target,  and  lethality  models  are  being 
integrated  into  a  modern  package,  which  will  make  the  data  more 
readily  available  to  users.  However,  the  impact  of  this  effort  is  likely 
to  be  limited  because  the  current  element  models  embed  those  data 
packages.  The  new  products  appear  to  be  useful  primarily  to  kinetic 
interceptors  (KI),  for  which  no  system  level  model  is  being  built.  A 
fundamental  limit  is  the  lack  of  data  at  appropriate  hypervelocity 
closing  velocities  with  large  masses,  which  cannot  be  addressed 
through  laboratory  or  ground  tests.  Thus,  ongoing  efforts  will  make 
the  data  more  accessible,  but  not  more  credible. 
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Findings  (cont.) 

■  IV&V  (and  A)  needs  to  go  well  beyond  “business  as  usual” 

-  The  credibility  of  the  BMD  system  performance  will  be  uniquely 
dependent  on  the  credibility  of  the  M&S  insight 

-  Need  assurance  that  projections  inferred  represent  the  best 
Insight  that  can  be  offered 

■  Peer  review  by  “outsiders”  is  required  to  assure  the  fidelity 
of  the  scientific  and  engineering  representations 

-  Not  currently  accomplished  oonsistently  across  the  program 

■  The  Nuclear  Quality  Assurance  (NQA)  approach  may  be 
required  to  document  all  scientific  and  engineering  M&S 
representations  and  to  enable  traceable  configuration 
control 


A  key  element  of  configuration  control  is  independent  verification 
and  validation  (IV  &  V)  (and  accreditation  [A]).  The  Capability 
Maturity  Model  (CMM)  level  designation  provides  a  measure  of  the 
maturity  and  discipline  inherent  in  a  software  development  team. 

The  nuclear  reactor  community  (submarine  and  civilian)  has  a  rigid 
nuclear  quality  assurance  NQA  regimen.  All  calculations  are 
documented  in  scientific  notebooks;  all  are  checked.  Use  of 
unchecked  calculations  or  codes  is  simply  not  permitted.  This 
regimen,  foreign  to  MDA,  would  introduce  immediate  configuration 
control  for  spiral  development  and  block  change.  In  addition,  it 
would  facilitate  IV&V  and  accreditation  processes. 
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Findings  (cont.) 


■  Area  1  -  general  scope  and  content  quality  of  W&A 

-  The  MDA  VV&A  process  has  trended  over  several  years  (since  SDI)  from 
“head  to  head”  comparison  of  parallel  models  toward  use  of  fewer 
models,  each  checked  by  a  relatively  smaller  "associated  but 
independent”  VV&A  effort 

-  The  impact  of  this  trend  will  increase  beyond  Block  04  as  fewer  of  the  key 
models  have  benefited  from  parallel  comparison 

■  Area  2  -  process  and  scheduling  rigor  of  W&A 

-  MDA  has  substantially  increased  emphasis  on  M&S  VV&A  processes 

•  Importance  of  incorporating  and  maintaining  established  software 
practices  including  W&A  reflected  by  management  personnel 
qualifications,  M&S  directives  and  W&A  program  content 

-  However,  diversity  of  VV&A  process  description  details  by  the  element, 
program  and  core  W&A  presentations  raises  concern 

-  Proprietary  M&S  codes  must  not  be  permitted  to  impede  IVV&A 
-  Overall.  W&A  comprises  -10-15%  of  total  M&S  funding 


Regarding  the  general  method  by  which  missile  defense  VV&A 
seeks  to  ensure  M&S  physics  and  algorithm  quality,  area  one  in  the 
discussion  above,  the  Task  Force  observes  the  following  trend.  In 
earlier  years  it  was  common  practice  to  develop  and  compare 
multiple  and  largely  independent  models.  Such  "parallel"  models 
may  have  initially  incorporated  their  own  sets  of  implicit  and  explicit 
assumptions,  but  final  results  were  vetted  by  give  and  take  among 
multiple  M&S  development  teams  expert  in  the  phenomenology, 
physics,  and  engineering  areas  involved.  This  practice  has  generally 
given  way  to  one  in  which  significantly  fewer  core  and  element 
models  are  relied  upon  for  predicting  performance  parameters. 
VV&A  is  now  generally  performed  by  a  number  of  comparatively 
small  groups,  one  being  assigned  to  each  major  model.  Whether  or 
not  this  situation  is  sufficient  to  ensure  M&S  content  quality  as 
overall  BMDS  complexity  increases  beyond  Block  04  will  be  further 
discussed  below. 

With  regard  to  W&A  process  specification,  configuration  control, 
and  scheduling  rigor,  area  two  discussed  above,  the  Task  Force 
observes  a  substantial  increase  in  emphasis  by  MDA  over  the  past 
two  years,  particularly  following  release  of  the  Weyant  M&S  report. 
The  positive  steps  taken  include  addition  of  management  personnel 
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well  versed  in  software  processes  including  W&A.  Also 
encouraging  is  the  W&A  direction  contained  recently  revised  M&S 
policy.  However,  from  the  briefings  received  it  is  also  observed  that 
improvements  in  policy  have  not  yet  fully  translated  to  uniform 
W&A  process  description  and  practices  across  all  the  element  and 
core  models.  Accreditation  status  and  currency  is  also  not  yet 
uniform;  within  a  single  simulation  it  is  common  to  find  some  models 
accredited  and  some  not.  The  fact  that  some  key  models  are  held  as 
proprietary  to  their  developers  was  noted,  although  it  is  not  clear 
whether  or  not  this  creates  any  impediments  to  the  W&A  process. 

Lastly,  it  was  observed  that  the  appropriate  level  of  resources 
currently  being  allocated  to  W&A  represents  10-15  percent  of  the 
amount  dedicated  to  M&S  overall. 
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Findings  -  Area  1 

General  Scope  and  Content  Quality  of  VV&A 

■  For  Block  04,  non-C2/BM/C  W&A  is  generally  sufficient,  largely 
owing  to  long  maturation  of  key  models,  although  some  aspects,  e.g., 
track  redundancy  and  cuing,  warrant  particularly  close  continued 
W&A  attention 

■  For  Block  04,  VV&A  for  any  required  new  C2/BM/C  M&S  efforts  are 
'  particularly  hard  pressed  by  the  deployment  schedule 

■  For  blocks  beyond  04,  current  VV&A  of  “physics/algorithms”  M&S 
critical  to  prediction  of  future  element  and  associated  architecture 
performance  may  be  Insufficient,  e.g.,  Kl  and/or  SSTS 

■  Level  of  W&A  for  blocks  beyond  04  should  reflect  increasing  BMDS 
and  associated  M&S  complexity,  particularly  in  the  C2/BM/C  area; 
IW&A  level  of  support  may  be  Insufficient  to  do  so 


The  IW&A-related  conclusions  may  be  categorized  into  the  same 
areas  discussed  above.  For  area  one,  dealing  with  the  M&S  technical 
quality,  the  Task  Force  concludes  that  Block  04  models,  with  the 
notable  exceptions  of  those  for  C2/BM/  C  and  perhaps  multi-element 
functions  such  as  track  correlation  and  weapon  cuing,  have 
effectively  been  verified  and  validated  for  their  intended  Block  04 
uses.  These  models  benefit  from  much  prior  use  and  a  generally 
greater  degree  of  cross  checking  with  parallel  models  -  in  some  cases 
going  back  for  decades. 

This  general  endorsement  does  not  apply  to  the  C2/BM/C  area, 
however.  In  that  area,  there  is  a  concern  that  time  is  very  short  to 
perform  IW&A  for  new  models  such  as  MDWAR  sufficient  to 
reliably  predict  system  capabilities  and  limitations.  (Although  the 
Block  04  element  models  are  generally  sufficient  to  address  a  wide 
variety  of  questions,  it  is  impossible  to  anticipate  in  advance  all 
element  interactions;  because  of  this,  it  is  important  to  review 
carefully  the  core  models  for  each  use  in  making  program  decisions.) 

Beyond  Block  04,  there  are  additional  difficulties.  Many  future 
system  elements— such  as  KI  and  SSTS— are  still  at  immature  levels 
of  development,  use,  and  W&A.  This  conclusion  extends  to  a 
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particularly  serious  concern  in  the  case  of  C2/BM/ C  models  for 
future  blocks,  where  increasingly  numerous  systems  interactions 
imply  increasingly  complex  VV&A.  Reliable  C2/BM/ C  modeling  is 
required  sooner,  not  later,  to  optimize  future  block  architectures  and 
designs  taking  full  account  of  ^reat  and  countermeasure  spectra. 
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Findings  -  Area  2 

Process  and  Scheduling  of  VV&A 

■  Additional  M&S  policy  and  directives  (post  2000),  along  with  strong 
management/staff  M&S  procedural  competency,  is  making  positive 
impact  on  W&A  process/control  rigor  and  uniformity 

■  No  master  “accreditation  tree”  with  clearly  delineated  buyoff  (and 
buyoff  recommendation)  dates  by  M&S  users,  including  the  two 
national  teams,  for  blocks/  spirals/  phases,  is  currently  approved 
(although  aspects  are  In  work).  Such  a  document  would  help  tie 
W&A  to  the  complex  BMDS  design/deployment  process  and  prevent 
M&S  “escapes” 


Overall,  given  the  architectural,  technological  and  operational 
complexities  of  BMDS,  a  W&A  level  of  effort  of  '■-1 0-1 5%  of 
total  M&S  (characteristic  of  M&S  for  much  less  complex 
systems)  should  be  closely  examined  for  sufficiency. 


Turning  to  area  two,  W&A  process  management,  the  Task  Force 
first  concludes  that  M&S  policy  and  directives  issued  after  2000, 
along  with  added  staff  M&S  procedural  competency,  is  making  a 
positive  impact  on  W&A  process  and  control  rigor  and  uniformity. 
(For  example,  the  basis  of  confidence  documents  now  being 
developed  for  the  element  and  core  simulations  provide  good 
documented  support  for  the  approval  of  models;  their  use  is 
encouraged.)  However,  accreditation  (participation,  uniformity, 
tracking,  and  synchronization  to  development/ deployment 
schedules)  is  found  to  be  an  area  for  improvement.  An  accreditation 
tree  incorporating  more  detailed  planning,  wider  distribution  for 
feedback,  and  regular  updating  is  thus  recommended  below. 

It  is  important  that  the  system  element  material  developers 
assume  a  stake  in  the  models  used  to  represent  the  system  elements 
developed  under  their  authority.  The  material  developers,  the  PMs, 
their  staff,  and  their  contractors,  are  most  familiar  with  the  details  of 
their  system.  They  are  most  familiar  with  the  key  engineering 
aspects  of  their  systems,  and  they  will  know  the  sources  of  error  and 
imperfection.  They  will  be  also  able  to  provide  informed  judgments 
regarding  degree  of  abstraction  uncertainty  and  are  likely  to  know 
about  the  idiosyncrasies  of  their  system.  Additional  M&S  policies 
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aiid  managerial  directives  are  anticipated  to  make  a  positive  impact 
on  needed  participation  in  accreditation. 

A  final  conclusion  that  overarches  both  the  technical  quality  and 
the  process  management  areas  is  that,  given  the  architectural, 
technological,  and  operational  complexities  of  BMDS,  a  W&A  level 
of  effort  of  ~10-15  percent  of  total  M&S  (characteristic  of  M&S  for 
much  less  complex  systems)  should  be  closely  examined  for 
sufficiency.  (This  statement  may  apply  to  W&A  of  operational 
codes,  but  these  were  not  examined  by  the  Task  Force.) 
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Recommendations 

■  Form  an  IW&A  Quality  Assessment/Audit  Team  with  a 
balance  of  experienced  MD  element  algorithm,  C2/BM/C 
and  software  experts,  as  well  as  representatives  from  the 
M&S  user  and  MD  operator  communities 

-  Report  directly  to  MDA  M&S  management  to  identify  any  escapes 
in  IW&A  uniform  technical  quality  and  standard  process 
compliance 

-  Critique  IVV&A  milestone  dates  in  view  of  block  development 
schedules 

-  Make  recommendation  on  all  accreditations 

-  Coordinated  with,  but  smaller  than  and  independent  of  current 
M&S  IPT 


Based  on  the  above  considerations,  findings,  and  conclusions,  the 
DSB  Task  Force  has  formulated  four  recommendations  in  the  area  of 
VV&A.  The  first  two  are  aimed  at  ensuring  the  technical  quality  of 
the  W&A  process;  the  latter  two  deal  with  process  and  funding 
concerns. 

A  relatively  small  standing  assessment/ audit  team  should  report 
directly  to  senior  MDA  M&S  management  to  identify  any  escapes  in 
VV&A  uniform  technical  quality  and  process  compliance,  and  would 
augment  the  strong  process  orientation  of  the  current  management 
team.  This  team  should  critique  VV&A  milestone  dates  in  view  of 
the  block  development  schedules  and  help  to  harmonize  tlie  technical 
status  of  the  models  with  the  need  dates.  It  is  further  suggested  that 
this  team  make  recommendations  on  future  accreditations.  The  team 
would  coordinate  with  and  not  take  the  place  of  the  larger  M&S 
Integrated  Product  Team  (IPT)  currently  in  place. 


57 


Missile  Defense  Phase  in 


Ensuring  the 
Validity  of  Simula  tjons 


Recommendations  (cont.) 

•  Further  refine  and  document  IVV&A  strategy  to  ensure 

M&S  validity  to  the  many  “envelope  corners”  of  global 

multi-tiered  BMD 

-  Pay  attention  to  what  is  not  included  in  the  models,  to  what  is 
assumed  to  work  perfectly,  and  to  the  sensitivity  of  the  results  to 
uncertain  input  parameters 

-  “Break  the  codes"  in  as  many  ways  as  possible  (a  few  examples  - 

ROEs,  shot  correlation,  penaids,  track  branching . ) 

-  Expand  “threads"  strategy  to  check  all  functionality  of  all  models 

-  IW&A  the  data  catalogs  for  all  models  to  ensure  that  key 
variances  and  other  statistics  are  recorded  and  tracked 


This  recommendation  is  to  help  ensure  that  M&S  can  stand  as  a 
principal  source  of  confidence  in  the  missile  defense  systems 
performance  over  the  entire  range  of  expected  operating  conditions. 
VV&A  must  identify  what  gaps  exist  in  the  models  as  well  as  what  is 
being  erroneously  assumed  to  work  perfectly.  The  strategy  should  be 
to  "break  the  codes"  in  as  many  ways  as  possible. 

Examples  of  this  strategy  include  testing  tlie  models  under 
multiple  threat-defense  geometry  and  saturation  scenarios  and  under 
various  countermeasure,  shot- to-shot  correlation,  and  track 
branching  conditions.  One  relatively  standard  tool  that  the  strategy 
should  consider  is  the  use  of  carefully  designed  code  testing 
"threads"  to  stress  all  code  functionality.  In  addition,  it  is  important 
that  the  W&A  process  examine  the  parametric  data  used  in  all 
models  with  regard  to  accuracy  and  completeness,  as  well  as  include 
all  variances  necessary  for  purposes  such  as  Monte  Carlo  runs.  The 
mechanisms  for  updating  and  tracking  parameter  and  variance 
values  based  on  hardware  testing  should  also  be  included. 
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■  Develop,  or  accelerate  development  of,  accreditation  tree 
for  all  models,  but  especially  for  higher  level  models,  that 
addresses  previously  stated  concerns.  Approvals  should 
probably  include  that  of  NTS,  and  perhaps  NTB,  but 
should  definitely  culminate  at  a  higher  government  levei 

■  MDA  should  review  IW&A  total  level  of  effort  to  ensure 
that  it  supports  the  complexity  level  and  criticality  of  the 
M&S  effort  (to  address  previously  stated  observation  and 
conclusion  points) 


Accreditation  sign-offs  would  likely  include  that  of  NTS  and 
perhaps  NTB,  but  should  definitely  culminate  at  a  higher 
government  level.  This  process  recommendation  is  aimed  at 
establishing  better  uniformity  and  tracking  of  all  M&S  verification 
and  validity  responsibilities  and  current/ projected  status.  While  the 
VV&A  material  presented  in  MDA's  briefings  during  this  Task  Force 
study  did  indicate  that  there  was  an  accreditation  process  extant  for 
each  model,  these  separate  accreditation  processes  did  not  appear  to 
be  extremely  uniform  with  clear-cut  responsibilities,  sequencing,  and 
stakeholder  participation.  The  Task  Force  views  this 
recommendation  as  a  way  of  extending  further  the  existing  positive 
trend  toward  highest  quality  W&A  process  and  metrics. 

While  the  Task  Force  did  not  review  VV&A  funding  in  depth,  this 
recommendation  stems  mainly  from  the  previously  discussed 
observation  that  MDA  W&A  levels,  compared  to  the  levels  of  the 
supported  M&S,  appear  to  be  comparable  to  VV&A  levels  for  many 
less  complex  and  less  critical  (than  missile  defense)  DoD  systems. 

Finally,  while  not  a  formal  recommendation,  it  is  suggested  that 
MDA  discuss  VV&A  strategy  and  tactics  with  both  government  and 
integration  contractor  W&A  personnel  associated  with  the  Navy 


59 


Missile  Defense  Phase  III_ 


Ensuring  the 
Validity  of  Simula  tions 


Aegis  program.  Aegis  is  a  highly  complex  system  with  many  M&S 
parallels  to  ballistic  missile  defense.  A  brief  teleconference  with  M&S 
experts  at  the  integration  contractor  indicated  W&A  for  the  many 
Aegis  models  was  headed  by  the  Navy  (at  Dahlgren)  for  U.S. 
operational  ships,  and  by  the  integration  contractor  (at  Moorestown) 
for  foreign  sales  ships.  Both  structures  seem  to  work  well,  but  more 
importantly  there  appear  to  be  lessons  learned  of  use  to  MDA  in  both 
locations. 
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Recommendations  (cont.) 


■  MDA  should  assess  the  peer  review  approach  used  by 
the  DOE  Labs  as  a  basis  for  establishing  a  model  for  the 
FFRDCs  to  make  a  value  added  contribution  as  a  source 
of  IV&V  and  accreditation 

-  Depending  upon  the  results  of  the  assessment  (conducted  by  an 
independent  review  team),  MDA  should  then  select  and  implement 
a  FORMAL  configuration  control  and  IVV&A  approach 


Independent  review  is  an  important  aspect  of  adequate  W&A. 
Peer  review  is  used  effectively  for  this  purpose  in  the  Department  of 
Energy  (DOE)  laboratory  system.  While  there  is  no  complete  analog 
available  to  MDA,  it  may  be  that  one  or  more  Federally  Funded 
Research  and  Development  Centers  (FFRDCs)  could  fill  that  role. 
MDA  should  evaluate  the  potential  value  of  such  an  approach. 
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Chapter  5.  Additional  Opportunities  for  M&S 
Contribution 

The  Task  Force  recognized  that  additional  opportunities  exist  for 
modeling  and  simulation  in  the  development,  deployment,  and 
preparation  to  operate  a  ballistic  missile  defense  system.  As  a 
consequence,  issues  beyond  the  scope,  breadth,  fidelity,  element 
"communication,"  and  verification  were  considered. 


Findings  -  Scope  and  Integration 


■  Opportunity  exists  to  provide  M&S  support  to: 

-  Develop  the  defined  Block  of  capability  and  to  evaluate  and  assess  its 
component,  element,  and  integrated  system  performance 

-  Explore  trade-offs  for  evolution  towards  the  end  capability  -  an  Integrated 
missile  defense  to  defeat  missiles  of  all  ranges  in  all  phases  of  flight 

■  Current  M&S  directed  at  supporting  1,  but: 

-  Current  system  level  architecture  approaches  rely  primarily  on  kinematic 
analyses  at  the  element  level 

-  Integrated  system  level  M&S  needs  to  offer  the  capability  to  assess  the 
merits  of  integrating  land,  sea,  and  space  based  resources 

-  The  need  to  assess  advantages  of  true  system-level  integration  or  true 
fusion  of  resources  has  not  been  adequately  served  by  separate,  and 
often  uncoordinated,  systems  level  and  C2/BM/C  M&S  efforts 


MDA  tends  to  rely  on  modeling  and  simulation  as  a  "tool"  and 
views  M&S  from  a  variety  of  perspectives  as  part  of  the  cost  of  doing 
business.  The  opportunity  exists  to  provide  coherence  and  rigor  to 
the  M&S  process  across  MDA  and  to  now  use  credible, 
independently  verified  and  accredited  M&S  techniques  to: 

1 .  Evaluate  the  capabilities  (needed)  for  each  element  of  a  given 
block 

2  .  Provide  a  basis  for  determining  and  defining  a  path  ahead 
based  upon  well  understood  tradeoffs  of  performance  vice 
capability  evolution,  schedule,  and  costs 
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Because  of  "grass  roots"  M&S  development  at  the  component, 
subsystems,  and  even  element  level  and  the  adoption  of  a 
"capabilities-based"  architecture  perspective,  there  has  been 
reluctance  to  use  M&S  to  "define"  systems-level  (including 
C2/BM/C)  evolution  paths. 
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Findings  -  Scope  and  Integration  (cont.) 


■  Need  to  also  focus  a  significant  M&S  effort  on  exploring  tradeoffs  for 
formulation  of  future  (Block  6  and  beyond)  architectures 

-  There  does  not  appear  to  be  a  coherent  vision  for  what  M&S  capabilities 
are  needed  so  that  future  system  evolution  paths  can  be  assessed 

»  Increased  attention  is  needed  for  M&S  to  also  support  developing  and 
implementing  a  long  term  vision  for  global  missile  defense 

-  BMDS  cannot  be  flight  tested  as  a  system  against  any  significant  portion 
of  the  “operating  envelope” 

“  M&S  validated  by  IV&V  and  systems  integration  “tests”  will  represent  THE 
SYSTEM  to  the  user 

-  The  users  -  STRATCOM  and  the  RCCs  -  do  not  have  the  resident 
capability  to  develop  models  of  BMDS  system  performance 

•  They  are  charged  with  planning  and  execution  with  assurance  to  the 
NCA  of  how  the  system  and  its  elements  will  perform 

•  BMDS  M&S  needs  to  include  support  for  this  warfighting  function 


Modeling  and  simulation  should  be  elevated  in  status  within 
MDA  so  that  it  is  viewed  as  the  deliverable  representation  of  "the 
system"  both  to  MDA  and,  importantly,  to  the  user. 

A  credible  M&S  representation  of  BMDS,  as  verified  in  limited 
test  bed  demonstrations,  will  provide  the  user  and  National 
Command  Authority  emulation  of  system-level  performance  over  all 
comers  of  the  envelope  against  adversary  threats  yet  to  be  developed. 

M&S  thus  needs  to  be  the  basis  for  the  formulation  of  Block  6  and 
subsequent  architectures.  Importantly,  if  used  with  rigor  across  both 
systems  and  C2/BM/C,  M&S  can  assure  that  evolution  paths  are  not 
precluded  for  future  BMDS  development. 
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Findings  -  Scope  and  Integration  (cont.) 

■  Integrated  system  effectiveness  -  achieving  the  intended  end 

capability  -  is  likely  to  also  depend  significantly  upon  offense/defense 
integration 

-  The  need  for  USSTRATCOM  and/or  RCCs  to  defend  areas  of 
responsibility  (AORs)  effectively  against  potential  threats  require 
preemptive  strikes  against  some  launch  areas 

•  While  not  an  assigned  MDA  mission,  the  M&S  program  needs  to  account  for  it 

•  MDA-developed  systems  will  be  the  primary  source  of  precision  data  for  attack 

-  STRATCOM,  to  improve  the  RCC  capability  to  defend  their  AORs,  must 
offer  a  global  missile  defense  perspective  for  the  benefit  of  all  and  provide 
the  command  and  control  services  necessary  to  support  global  missile 
defense 

->  M&S  of  the  BMDS  operation  will  be  the  basis  for  STRATCOM  and  the 
RCCs  to: 

•  Develop  contingency  plans  for  the  AORs  and  global  missile  defense 

•  Characterize  the  integration  of  offensive  and  defensive  forces 

•  Assess  the  likely  effectiveness  of  the  deployed  system  against  evolving 
threats 

-  The  BMDS  M&S  “deliverable”  must  be  credibly  integrated  into  force 
planning  and  execution  tools 


Change  2  to  the  Unified  Command  Plan  assigns  global  missile 
defense  responsibilities  to  USSTRATCOM.  To  fill  this  role, 
USSTRATCOM  and  the  regional  combatant  commands  will  need  a 
verified  capability  to  model  and  simulate  the  system's  behavior. 

Measures  of  effectiveness  of  missile  defense  will  include  area 
protected  (in  the  United  States,  or  regions  of  the  world  with  U.S. 
deployed  forces,  friends,  and  allies)  and  launch  area  denial.  The 
latter,  a  measure  of  the  threat  posed  from  regions  of  the  globe,  may 
well  only  be  countered  through  precision  strike  of  launchers  in  the 
area.  MDA-developed  systems  will  be  the  primary  source  of 
precision  data  for  attack. 

This  consideration  alone  is  rationale  for  strong  coupling  of 
offense  and  defense.  In  addition,  USSTRATCOM— with  global  strike, 
missile  defense,  and  global  intelligence,  surveillance,  and 
reconnaissance  (ISR)  integration  responsibilities— has  the  need  to  be 
able  to  use  elements  of  the  system  efficiently  and  effectively. 
Modeling  and  simulation  of  the  BMDS  should  account  for  these 
integration  objectives. 
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Effective  modeling  and  simulation  of  tlie  global  BMDS  concept  of 
operations  will  provide  USSTRATCOM  and  the  regional  combatant 
commanders  with  the  capability  to  develop  architecture  evolution 
plans,  understand  the  performance  of  the  system  in  spiral 
development,  and  plan  (and  train)  for  scenarios  of  the  future. 
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Findings  -  Scope  and  Integration  (cont.) 

■  Effective  M&S  support  of  block  evolution/system 
integration  phasing  within  spiral  development  approach 
demands  effective  M&S  configuration  control 

-  In  an  environment  of  day-to-day  changes,  “phased"  system 
integration  as  a  result  of  test  bed  evaluations,  spiral  development, 
and  block  changes  to  architecture  (capabiiity)  options 

■  Air  defense  is  not  part  of  the  current  MDA  M&S  evaluation 
task 

-  Severai  of  the  elements  (e.g.,  PAC-3,  Aegis)  have  air  defense 
responsibilities  and  obvious  self  defense  priorities 

-  The  systems  level  BMDS  M&S  must  include  representation  of  the 
resource  allocation  and  C2/BM/C  issues  resident  in  multi-mission 
elements  to  address  systems-level  performance 


The  entrepreneurship  of  code  developers,  the  evolution  of 
components,  subsystems,  and  elements  of  the  system  through  spiral 
development,  and  the  lack  of  systems  engineering  driven 
requirements  modus  operandi,  makes  "configuration  control"  of 
modeling  and  simulation  produce  a  formidable  set  of  challenges. 

Modeling  and  simulation  for  elements  of  the  system  and  for 
future  BMDS  evolution  do  not  reflect  the  capabilities  inherent  or 
essential  for  operation  in  many  of  the  elements.  For  example.  Aegis 
and  Pabiot  self-defense  objectives  against  cruise  missiles  are  not 
accounted  for  in  BMDS  models.  Thus,  the  resource  allocation  models 
viewed  to  be  available  for  ballistic  missile  defense  applications  are  in 
question. 
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Findings 

■  Modeling  &  simulation  evolution  towards  a  systems  level 
product  suitable  for  assessing  capabilities  and  making 
systems  trade-offs  for  future  spirals  appears  diffused, 
complicated,  and  slowed  by  the  split  between  systems 
level  and  C2/BM/C  M&S  developments 

■  BMDS  M&S  activities  need  to  fully  account  for  net-centric 
and  GIG-BE  evolution 

-  These  activities  need  to  provide  for  rapid,  collaborative  planning, 
command  and  control  support  for  the  engagement  environments 
facing  STRATCOM  and  the  RCCs 

-  Fusion  of  ISR  and  weapon  system  sensor  data  for  BMDS 
implementation  needs  to  be  explicitly  included  In  system  level 
M&S 


The  current  division  of  M&S  efforts  between  systems  and 
C2/BM/C  seems  detrimental  to  developing  an  effective  integrated, 
global  BMDS  (one  that  uses  multi  sensor  discrimination). 
Communications  initiatives  (NCES  and  GIG-BE)  have  been  largely 
ignored.  Thus,  data  fusion  and  global  interoperability  are  not 
modeled  (or  defined). 

BMDS  "global  system  level"  models  must  account  for  integrated 
elements  and  C2/BM/  C  representing  a  system-of-sy stems  in  order  to 
capture  how  the  system  "works"  and  how  it  can  be  used  against 
potential  threat  scenarios. 
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Findings  (cont.) 


■  Configuration  control  will  be  critical  in  developing  and 
using  the  models  and  simulations  for  future  spirals 

-  Definition  (or  identification)  of  a  standard  architecture  is  required 
for  each  biock  so  that  the  M&S  software  is  compatible 

-  The  “handover"  (modei  update)  for  subsequent  blocks  must  be 
managed  through  configuration  control 

"  IV&V  and  accreditation  is  currently  performed  on  certain 
(not  all)  BMDS  M&S  models  internally 

■  Two  relevant  models  are  the  DOE  Weapons  Lab  “peer 
review”  process  and  the  NQA  software  development 
standard  -  the  current  MDA  approach  does  not  appear  to 
meet  this  standard 

■  Internal  BMDS  M&S  IV&V  and  accreditation  does  not 
appear  to  have  the  organizational  stature  nor  resources 
for  the  task(s)  at  hand 


Modeling  and  simulation  does  not  appear  to  have  the  "status" 
within  MDA  to  achieve  the  needed  capability.  Budget, 
independence,  and  direct  reporting  responsibilities  to  MDA 
leadership  are  essential. 

MDA  needs  to  determine  a  standard  technical  architecture  for  the 
interface  of  the  M&S  block  components.  This  decision  should  be 
made  within  6  months  in  order  to  give  industry  clear  direction.  The 
high  level  architecture  (HLA)  is  currently  designated  by  DoD,  but 
recent  commercial-of-the-shelf  options  such  as  the  subscriber/ 
publish  software  should  be  considered  as  an  alternative.  A  small 
team  composed  of  people  with  relevant  software  experience  from 
government,  contractors,  and  FET^DOs  should  be  tasked  to  make 
recommendations  to  MDA.  After  making  a  recommendation,  MDA 
should  then  require  all  contractors  to  adhere  to  the  standard,  starting 
with  Block  2008  capabilities. 


DSB  Task  Force  ON 


70 


Chapters 


Recommendations 


■  View  systems  level  M&S  to  be  a  deliverable  to  USSTRATCOM 

-  On  behalf  of  STRATCOM  and  the  RCCs,  STRATCOM  should  offer  flag 
level  assignment  to  (at)  MDA  responsible  for  "buying”  (accepting)  the 
systems  level  BMDS  M&S  capability  for  the  user 

■  STRATCOM  should  use  the  combined  expertise  resident  at  the  JNIC 
and  STRATCOM  to  develop  an  integrated  offense/defense  M&S 
capability  based  upon  the  MDA  deliverable 

■  MDA  should  assign  a  senior  individual  responsibility  for  BMDS  M&S 
for  each  block 

-  That  individual  must  have  the  authority,  budget,  and  staff  to  integrate 
systems  and  C2/BM/C  M&S  efforts  into  a  credible  “BMDS  M&S 
deliverable"  for  each  block  to  include  responsibility  for  configuration 
control  and  utility  of  the  product  for  the  user 

■  MDA  should  assign  a  senior  Individual  to  lead  IV&V  and  accreditation 

-  To  report  independently  and  directly  to  Director,  MDA 

-  With  the  authority  and  budget  to  execute  the  desired  independent 
program 


Recommendations  on  this  and  the  next  two  charts  are  self 
explanatory. 


Recommendations  (cont.) 

■  Top  level  BMDS  M&S  tools  should  be  used  to  establish 
tradeoffs  for  future  architectures  based  upon  well 
understood  (common  fidelity)  systems  and  C2/BM/C 
representations 

■  BMDS  M&S  should  account  for  the  other  operational 
responsibilities  resident  in  the  elements  (e.g.,  air  defense, 
cruise  missile  defense)  so  that  the  operational 
effectiveness  of  the  system  as  a  whole  can  be  reliably 
assessed 

■  MDA  should  assess  utilization  of  the  NQA  approach  to 
force  configuration  control,  regimen,  and  appropriate  peer 
review  of  the  systems  level  BMDS  M&S  capability 
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Recommendations  (cont.) 


■  MDA  should  immediately  assign  a  small  team  of 
government  staff  supported  by  contractors  to  consider 
various  architecture  standards  (e.g.,  HLA)  with  the 
objective  of  recommending  a  BMDS  M&S  standard  within 
six  months.  MDA  should  promulgate  the  standard  and 
require  all  contractors  to  adhere,  starting  with  Block  2008 

■  MDA  should  integrate  the  systems  and  C2/BM/C  M&S 
approaches  so  that  a  useable  M&S  characterization  of 
BMDS  can  be  efficiently  and  effectively  produced 
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MEMORANDUM  FOR  CHAIRMAN,  DEFENSE  SCIENCE  BOARD 


SUBJECT:  Terms  of  Reference-Defense  Science  Board  Task  Force  on  Ballistic 
Missile  Defense  -  Phase  III  (Modeling  and  Simulation) 

In  coordination  with  guidance  from  the  Director,  Missile  Defense  Agency,  you  are 
requested  to  form  a  Defense  Science  Board  (DSB)  Task  Force  on  Ballistic  Missile 
Defense  —  Phase  III  (Modeling  and  Simulation). 

Modeling  and  simulation  (M&S)  plays  a  key  role  in  both  developing  the  Ballistic 
Missile  Defense  System  and  assessing  its  likely  effectiveness.  In  this  context,  modeling 
and  simulation  covers  the  gamut  from  detailed  engineering  simulation  of  components  to 
systems  of  systems  simulations. 

Specific  purposes  of  this  phase  of  the  task  force  are  to  assess: 

)  the  scope  of  the  modeling  and  simulation  effort 

2)  the  appropriateness  of  the  level  of  fidelity  of  classes  of  simulations 

3)  The  impact  of  communications  in  the  end-to-end  models 

4'  the  approaches  to  ensuring  the  validity  of  simulations  for  all  uses,  including 
exercises  and  wargaming  done  for  training  and  operations  concept 
development 

5)  additional  opportunities  for  M&S  contribution  to  BMDS  development  and 
evaluation 


The  Task  Force  is  asked  to  address  these  issues  in  a  system  of  systems  context  with 
particular  emphasis  on  battle  management  systems,  command  and  control  systems,  and 
the  global  sensor  system.  The  Task  Force  is  also  asked  to  provide  advice  on  the  state  of 
modeling  and  simulation  for  use  in  assessing  overall  performance  of  segments  of  the 
BMDS,  e.g.,  ground-based  midcourse  intercept  system,  space-based  interceptor  system. 


This  Task  Force  will  be  co-sponsored  by  me  as  the  Under  Secretary  of  Defense 
(Acquisition,  Technology,  and  Logistics)  and  the  Director,  Missile  Defense  Agency. 

Dr.  William  R.  Graham  and  General  Larry  D.  Welch,  USAF  (Ret)  will  serve  as  Task 
Force  Co-Chairmen.  Mr.  Paul  Hoff  will  serve  as  the  Executive  Secretary;  and  LtCol 
Roger  W.  Basl,  USAF,  will  serve  as  the  DSB  Representative. 

The  Task  Force  will  be  operated  in  accordance  with  the  provisions  of  P.L.  92-463, 
the  "Federal  Advisory  Committee  Act,”  and  DoD  Directive  5104.5,  "DOD  Federal 
Advisory  Committee  Management  Program.”  It  is  not  anticipated  that  this  Task  Force 
will  need  to  go  into  any  "particular  matters"  within  the  meaning  of  Section  208  of  Title  18, 
United  States  Code,  nor  will  it  cause  any  member  to  be  placed  in  the  position  of  acting  as 
procurement  official. 
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Abbreviations 


ABL 

Airborne  Lasers 

AOR 

Area  of  Responsibility 

BMC 

Battle  Management  and  Communications 

BMDS 

Ballistic  Missile  Defense  System 

BOC 

Basis  of  Confidence 

BPI 

Boost  Phase  Intercept 

C2 

Command  and  control 

C2/BM/C 

Command  and  Control/Battle  Management/Communications 

CAPS 

Commanders  Analysis  Planning  and  Simulation 

CEC 

Cooperative  Engagement  Capability 

CMM 

Capability  Maturity  Model 

CONOPS 

Concepts  of  Operations 

DIS 

Distributed  Interactive  Simulation 

DoD 

Department  of  Defense 

DOE 

Department  of  Energy 

DSB 

Defense  Science  Board 

EADSIM 

Extended  Air  Defense  Simulation 

EADTB 

Extended  Air  Defense  Test  Bed 

EDO 

Emergency  Defensive  Operations 

EKV 

Exoatmospheric  Kill  Vehicle 

EO 

Electro-Optic 

FFRDCs 

Federally  Funded  Research  and  Development  Centers 

GBI 

Ground-Based  Intercept 

GBR 

Ground  Based  Radar 

GMD 

Ground-Based  Midcourse  Defense 

HLA 

High  Level  Architecture 

HTK 

Hit-to-Kill 

HWIL 

Hardware-in-the-Loop 
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ICBM 

IDO 

IMD 

IMU 

IRBM 

IRT 

ISR 

IV&V 

IVV&A 

JNIC 

JPL 

JWARS 

JTWARS 

Kl 

KV 

M&S 

MDA 

MDAP 

MDSE 

MDWAR 

MRBM 

MSX 

NQA 

PAC  2/3 

PEELS 

PES 

RCC 

RF 

SBIRS 

SM-3 

SPO 

SPY-1 

SRBM 

SSGM 


Intercontinental  Ballistic  Missile 

Initial  Defensive  Operations 

Integrated  Missile  Defense 

Inertial  Measurement  Unit 

Intermediate  Range  Ballistic  Missile 

Independent  Review  Team 

Intelligence,  Surveillance,  and  Reconnaissance 

Independent  Verification  and  Validation 

Independent  Verification,  Validation,  and  Accreditation 

Joint  National  Integration  Center 
Jet  Propulsion  Laboratory 
Joint  Warfighting  System 
Joint  Theater  Warfighting  System 

Kinetic  Interceptors 
Kill  Vehicle 

Modeling  and  Simulation 
Missile  Defense  Agency 
Major  Defense  Acquisition  Program 
Missile  Defense  System  Exerciser 

Missile  Defense  Warfare  Assessment  and  Research  Simulation 
Medium  Range  Ballistic  Missile 
Midcourse  Space  Experiment 

Nuclear  Quality  Assurance 

PATRIOT  Advanced  Capability,  Ievel2/level  3 
Parametric  Endo-Exo  Lethality  Simulation 
Probability  of  Engagement  Success 

Regional  Combatant  Commands 
Radio  Frequency 

Space  Based  Infrared  System 
Standard  Missile-3 
System  Program  Office 
AEGIS  Radar 

Short  Range  Ballistic  Missile 
Strategic  Scene  Generation  Model 
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Appendix  C 


STSS 

Space  Tracking  and  Surveillance  System 

T&E 

Test  &  Evaluation 

THAAD 

Theater  High  Altitude  Area  Defense  System 

UEWR 

Upgrade  Early  Warning  Radar 

USD(AT&L) 

Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics 

USSTRATCOM 

United  States  Strategic  Command 

W&A 

Verification,  Validation,  and  Accreditation 

XBR 

X-Band  Radar 
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